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ABSTRACT 


The  problem  of  recognition  of  motion  concepts  from  a  sequence  of  images  is 
discussed  and  a  framework  for  its  solution  is  proposed.  This  framework  is  based  on 
several  of  the  current  paradigms  of  Artificial  Intelligence:  knowledge-based  recogni¬ 
tion,  hypothesize-and-test,  and  competition  and  co-operation  among  local  hypotheses. 
In  particular,  a  representation  for  motion  concepts  is  presented  that  is  based  on  se¬ 
mantic  networks  and  on  the  PSN  formalism.  A  control  structure  is  described  which 
uses  this  knowledge  base  as  an  aid  to  recognition  of  the  motion  concepts  exhibited  over 
a  sequence  of  images.  This  control  structure  uses  the  conceptual  adjacency  of  hy¬ 
potheses  in  conjunction  with  relaxation  labelling  and  similarity  links  between  hy¬ 
potheses  in  order  to  decide  which  hypothesis  best  matches  the  data  obtained  from  the 
image  sequence.  Hypothesis  similarities  and  time  course  of  differences  guide  the 
change  of  attention  mechanism,  while  system  inertia  is  the  basic  principle  used  by  the 
focus  of  attention  mechanism. 

Empirical  results  have  lead  to  the  design  of  a  temporal  segmentation 
mechanism  and  to  the  discovery  of  the  system  limitations.  These  limits  define  the 
shortest  recognizable  motion,  provide  an  empirical  rule  for  the  selection  of  adequate 
scene  sampling  rates,  and  define  the  conditions  under  which  a  temporal  segmentation 
process  is  necessary.  In  addition,  a  noise  sensitivity  parameter  is  provided.  Testing 
has  also  revealed  that  strength  of  competition  and  co-operation  between  hypotheses  is 
tunable  and  directly  related  to  the  domain  in  semantically  meaningful  ways. 

The  framework  is  being  realized  by  a  system  called  ALVEN,  whose  structure 
and  operation  are  described.  ALVEN  is  a  system  for  classification  of  motions  exhibited 
by  the  human  left  ventricle  in  the  course  of  a  heart  beat.  It  employs  a  knowledge  base, 
using  the  representational  formalism  defined  in  this  document,  that  contains  many  of 
the  current  hypotheses  for  left  ventricular  wall  motion  that  cardiologists  have  put  for¬ 
ward.  The  purpose  behind  this  system  is  to  provide  an  evolving  research  prototype  for 
experimenting  with  the  analysis  of  certain  classes  of  bio-medical  imagery,  and  for 
refining  and  quantifying  the  body  of  relevant  medical  knowledge. 
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CHAPTER  1 


INTRODUCTION 


" Next  for  discussion  after  the  subjects  mentioned  is  Time.  The  best  plan 
will  be  to  begin  by  working  out  the  difficulties  connected  with  it,  making  use  of 
the  current  arguments.  ... 

Some  assert  that  it  is  (1 )  the  movement  of  the  whole,  others  that  it  is  (2)  the 
sphere  itself.  ... 

But  as  time  is  most  usually  supposed  to  be  motion  and  a  kind  of  change,  we  must 
consider  this  view.  ... 

Not  only  do  we  measure  the  movement  by  the  time,  but  also  the  time  by  the  move¬ 
ment,  because  they  define  each  other.  ... 

It  is  natural  that  this  should  happen;  for  the  movement  goes  with  the  distance 
and  the  time  with  the  movement,  because  they  are  quanta  and  continuous  and 
divisible.  ... 

Since  time  is  the  measure  of  motion,  it  will  be  the  measure  of  rest  too  -  indirectly. 
For  all  rest  is  in  time.  ... 

These  distinctions  having  been  drawn,  it  is  evident  that  every  change  and  every¬ 
thing  that  moves  is  in  time;  for  the  distinction  of  faster  and  slower  exists  in 
reference  to  all  change,  since  it  is  found  in  every  instance.  In  the  phrase  "mov¬ 
ing  faster "  I  refer  to  that  which  changes  before  another  into  the  condition  in 
question,  when  it  moves  over  the  same  interval  and  with  a  regular  movement; 
e.g.,  in  the  case  of  locomotion ,  if  both  things  move  along  the  circumference  of  a 
circle,  or  both  along  a  straight  line,  and  similarly  in  all  cases.  But  what  is  "be¬ 
fore"  is  in  time;  for  we  say  " before "  and  "after"  with  reference  to  the  instance 
from  the  "now”,  and  the  "now"  is  the  boundary  of  the  past  and  the  future;  ... 

It  is  also  worth  considering  how  time  can  be  related  to  the  soul;  and  why  is  time 
thought  to  be  in  everything,  both  in  earth  and  in  sea  and  in  heaven.  Is  it  because 
it  is  an  attribute,  or  state,  or  movement  and  all  these  things  are  movable,  and 
time  and  movement  are  together  both  in  respect  of  potentiality  and  in  respect  of 
actuality?  ... 


Aristotle 

Physics,  Book  IV,  Chapters  10-14 
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Aristotle  wrote  Book  IV  of  Physics  roughly  2300  years  ago.  Even  then,  the 
great  thinkers  had  realized  that  motion  and  time  are  inextricably  related,  that  move¬ 
ment  is  defined  by  the  distance  traveled  and  that  time  measures  the  duration  of  travel. 
Aristotle  also  realized  that  in  order  to  compare  one  motion  against  another,  there 
must  be  a  point  of  reference:  that  the  objects  whose  motions  are  being  compared  must 
exhibit  similarities  of  motion  patterns  or  else  the  comparison  is  meaningless.  He  even 
gives  his  own  "frame-like”  definition  of  the  motion  concept  "faster". 

Twenty-three  hundred  years  later,  we  once  again  find  ourselves  concerned 
with  these  concepts  because  we  now  wish  to  have  computers  use  them  as  well.  Time 
and  motion  play  a  crucial  part  in  our  lives  -  both  evolutionary  and  everyday.  There  is 
no  need  to  discuss  the  importance  of  motion  and  time.  One  need  only  consider  the 
everyday  occurrances  -  making  appointments,  crossing  the  street  in  rush  hour,  timing 
a  left  hand  turn  in  the  face  of  oncoming  traffic,  and  so  many  more.  We  all  seem  to  do 
quite  well  in  the  face  of  such  adversities.  And  yet,  try  to  quantify  the  judgements  that 
we  make  in  order  to  succeed  at  such  tasks  and  we  run  into  seemingly  insurmountable 
problems. 

Computer  vision  research  has,  in  the  main,  concentrated  on  the  analysis  of 
isolated  images.  Scene  analysis  is  the  term  used  to  describe  the  process  by  which  the 
objects  in  a  single  image  are  extracted,  classified  and  related  to  one  another.  For  ex¬ 
ample,  in  a  system  that  analyzes  office  scenes  [Tenenbaum77],  a  typical  scene  descrip¬ 
tion  could  be  that  a  desk  is  in  the  center  of  the  room,  a  chair  is  behind  the  desk,  a  tele¬ 
phone  is  supported  by  the  desk  and  is  in  the  far  left  corner  of  it,  a  wall  is  behind  the 
chair  and  contains  a  door  and  a  painting.  Suppose  now  that  a  person  is  added  to  the 
scene  who  raises  the  telephone  receiver,  but  drops  it  while  the  image  is  being  record¬ 
ed.  The  resultant  image  is  ambiguous  for  the  system  described  above.  The  telephone 

V 

receiver  does  not  exhibit  a  support  relation,  and  in  static  images,  the  system  knows 
that  all  objects  must  be  supported  somehow.  There  are  a  multitude  of  similar  situa¬ 
tions  that  lead  to  ambiguous  single  images. 
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The  point  here  is  that  only  a  restricted  class  of  images  can  be  analyzed  out  of 
context  -  that  is,  without  considering  the  pragmatics  of  the  situation.  Just  as  in  natur¬ 
al  language  understanding,  where  the  context  in  which  a  sentence  is  found  aids  in 
disambiguating  its  meaning,  so  here  too,  the  motions  of  the  objects  can  aid  in  the 
analysis  of  images  in  which  temporal  information  is  present.  In  other  words,  the  in¬ 
terpretation  of  an  image  that  is  part  of  a  sequence,  can  be  facilitated  by  the  context 
represented  by  the  sequence. 

If  image  sequences  are  to  be  considered,  the  concepts  present  in  systems 
such  as  in  [Tenenbaum77],  (mainly  static  spatial  relations),  for  example,  or  in  any  oth¬ 
er  similar  system,  must  be  extended  into  the  time  domain.  In  order  to  do  this,  con¬ 
cepts  must  be  defined  that  relate  changes  of  static  relationships  in  time.  These  con¬ 
cepts  provide  the  context  within  which  individual  images  may  be  analyzed  -  the  prag¬ 
matics  of  the  vision  process. 

This  research  is  an  attempt  at  "uncomplicating"  time  and  motion.  The  con¬ 
cepts  of  time  and  motion  are  most  clearly  evident  in  movies  -  representations  of  mo¬ 
tions  in  the  real  world.  Therefore,  sequences  of  images  will  be  used  as  the  test  bed  for 
our  hypotheses  about  motion.  In  particular,  a  very  small  but  rich  domain  will  be  used  - 
that  of  human  left  ventricular  (LV)  wall  motion.  The  motions  will  be  represented  by 
cineangiograms.  These,  briefly,  are  X-ray  films  of  the  human  heart  beating  in  a 
patient's  chest.  They  •will  be  more  completely  described  later. 

This  domain  was  chosen  for  several  reasons: 

•  Any  success  in  the  research  would  be  very  valuable  to  the  medical  community. 

The  Cardiovascular  Unit  at  Toronto  General  Hospital  is  very  interested  in  any  such 
results  and  is  providing  both  high  quality  data  for  the  research  and  expert  medical  ad¬ 
vice  on  the  problem.  v 

•  The  left  ventricle  exhibits  many  of  the  concepts  important  for  motion  and  time 
analysis:  a  large  number  of  possible  motion  classes  for  each  segment  of  the  LV,  super¬ 
imposed  motions,  overlapping  timing  constraints,  ranges  of  acceptable  values  for  im- 
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portant  parameters,  and  difficult  edge  determination  in  the  images  so  that  guidance  is 
necessary. 

•  The  domain  is  understood  well  enough  so  that  a  knowledge  base  (KB)  of  LV  mo¬ 
tion  concepts  can  be  constructed. 

Our  research  has  impact  in  two  major  research  areas  -  artificial  intelligence 
and  medical  science.  The  primary  objective  is  the  advancement  of  design  methodolo¬ 
gies  for  computer  systems  that  analyze  motion.  A  second,  and  equally  important  ob¬ 
jective,  is  the  development  of  a  computer  system  that  can  perform  analysis  of  left  ven¬ 
tricular  wall  motion  by  processing  cinecardioangiograms.  Such  a  system  can  serve  as 
a  formal  model  of  the  mechanics  of  left  ventricular  wall  motion  and  also  as  an  assistant 
to  physicians  who  wish  to  both  reduce  the  amount  of  work  necessary  to  carry  out  a  di¬ 
agnosis  and  to  make  the  process  of  analysis  more  consistent  and  objective.  In  addi¬ 
tion,  such  formal  models  do  not  exist  at  the  present  time;  indeed,  hypotheses  about 
left  ventricular  dynamics  are  only  now  evolving,  and  a  system  such  as  ours  could  be 
used  as  a  testbed  for  such  hypotheses. 

Our  thesis  is  that  a  system  whose  control  structure  is  based  on  competition 
and  co-operation  among  hypotheses,  and  that  is  driven  by  the  organizational  relation¬ 
ships  of  a  knowledge  base,  is  adequate  (but  perhaps  not  the  only  solution  to  the  prob¬ 
lem)  for  classification  of  a  restricted  set  of  motions  exhibited  in  an  image  sequence. 
"We  will  call  such  a  system  a  Motion  Understanding  System  (MUS).  In  addition,  we 
claim  that  the  representational  formalism  we  have  defined  for  the  knowledge  base  is 
sufficient  for  representing  the  motion  concepts  necessary  for  general  motion  under¬ 
standing,  and  in  particular,  for  left  ventricular  wall  motion.  The  formalism,  however,  is 
restricted  in  that  it  cannot  represent  causal  relationships  between  motions,  and  it  only 
deals  with  two-dimensional  motions  and  therefore  occlusion  is  not  handled.  Extensions 

v 

are  left  for  the  future. 

The  KB  must  contain  both  general  and  problem-specific  motion  concepts.  The 
recognition  algorithms  use  this  KB  in  order  to  assist  them  in  determining  which  motion 
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classes  are  exhibited  by  objects  in  an  image  sequence.  Several  assumptions  are  neces¬ 
sary  for  the  recognition  control  structure: 

•  The  motions  in  the  image  sequence  are  continuous,  i.e.,  there  are  no  random 
changes  in  the  direction  or  magnitude  of  the  observable  inter-image  displacements. 

•  Each  motion  exhibited  occurs  over  several  images  (perhaps  necessitating  a  high 
scene  sampling  rate).  We  will  later  describe  empirical  results  that  aid  in  selection  of 
reasonable  sampling  rates. 

•  We  assume  that  a  conceptual  description  of  the  first  image  is  given  and  that  an 
initial  assignment  of  motion  hypotheses  to  each  object  is  done.  The  conceptual 
description  is  assumed  correct;  the  motion  hypotheses  on  the  other  hand,  may  be  to¬ 
tally  erroneous  and  mis-leading.  These  are  simplifications  for  the  purposes  of  our 
study  of  the  motion  aspects  of  image  sequences.  In  this  way,  the  system  start-up  prob¬ 
lem  is  overcome. 

•  As  in  other  knowledge  based  understanding  systems,  we  assume  that  the  KB  has 
predictive  power.  Information  in  the  KB  about  the  problem  domain  can  be  used  in  ord¬ 
er  to  predict  object  locations  in  each  image,  and  to  predict  possible  motion  classes  ex¬ 
hibited  by  each  object. 

It  is  important  to  define  the  limits  that  we  impose  on  our  methodology.  We 
assume  that  such  a  system  will  be  used  to  do  question-answering  about  the  films  it  is 
presented.  Questions  are  asked  before  the  sequence  is  analyzed  and  replies  come  only 
after  the  entire  sequence  has  been  analyzed.  For  example,  in  an  airport  situation,  a 
user  may  ask  "Is  the  airplane  rising?"  and  expect  to  receive  a  yes  or  no  answer.  If  the 
answer  is  no,  then  one  of  several  situations  may  have  occurred.  If  the  plane  was  initial¬ 
ly  rising,  and  then  changed  trajectory  in  some  way,  the  user  can  expect  to  be  given  this 
information.  Also,  if  the  plane  was  not  rising  at  all,  but  doing  sofnething  else  entirely, 
then  the  user  can  expect  to  be  told  that  the  plane  is  exhibiting  some  other  trajectory 
type.  There  is  no  mechanism  at  present  in  the  system,  however,  that  would  determine 
information  more  specific  than  the  user's  original  request.  That  is,  the  reply  will  be  ei- 
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ther  at  the  same  level  of  abstraction  or  at  a  more  general  one  -  never  at  a  more 
specific  one.  The  control  structure  does  not  preclude  such  a  mechanism,  and,  from  a 
human  engineering  viewpoint,  it  would  be  a  desirable  feature,  and  is  thus  left  for  future 
work. 

Our  methodology  integrates  several  current  Artificial  Intelligence  techniques 
with  several  novel  ones.  A  representation  formalism  for  general  temporal  events  is 
presented.  This  formalism  is  intended  to  handle,  all  of  the  semantic  components 
deemed  necessary  for  adequate  motion  description  in  two  dimensions,  but  with  the  ex¬ 
ception  of  causality  [Miller72].  Knowledge  units  (frames)  are  defined  for  each  motion 
concept  and  are  organized  using  concepts  from  semantic  network  theory  [Brach- 
man79],  particularly  the  PSN  formalism  [Levesque79].  An  intermediate  language,  using 
this  formalism,  is  provided  for  linking  quantitative  image  data  with  qualitative  concep¬ 
tual  information.  Using  our  representation  formalism,  a  knowledge  base  of  general  mo¬ 
tion  concepts  is  defined.  In  addition,  a  problem-specific  knowledge  base  for  left  ven¬ 
tricular  wall  motion  is  presented  which  uses  general  motion  concepts  for  motion 
description.  See  Chapters  3,  4  and  5  for  further  discussion  and  examples. 

The  paradigms  of  competition  and  co-operation  among  hypotheses  and 
hypothesize-and-test  form  the  basis  of  our  recognition  control  structure.  The  key 
feature  of  the  control  structure  is  that  it  is  driven  by  the  organization  of  the  knowledge 
base,  that  is,  by  the  primitive  relations  between  knowledge  units.  A  feedback  loop  is  in¬ 
corporated  in  order  to  link  the  several  components  of  the  control  structure.  The  low 
level  vision  process  is  expectation  guided,  with  modifications  to  the  relaxation  labelling 
process  of  [Zucker77]  integrating  high  level  expectations  with  edge  determination. 
Changes  in  object  positions  between  a  pair  of  consecutive  images  are  described  using 
the  intermediate  level  language  mentioned  above.  These  changes  are  matched  against 
hypothesized  motions  and  differences  are  detected.  The  characteristics  of  the 
differences  detected  aid  in  the  selection  of  competing  hypotheses.  Since  several  simul¬ 
taneous  hypotheses  can  co-exist,  a  focus  of  attention  mechanism  is  necessary  in  order 
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to  limit  the  number  of  hypotheses  under  consideration.  Our  focus  of  attention 
mechanism  uses  a  property  of  feedback  systems:  inertia.  That  is,  the  output  of  the 
system  changes  slowly  over  time  and  therefore  changes  in  focus  are  continuous  and  do 
not  shift  erratically.  Certainty  factors  are  attached  to  each  hypothesis  and  are  updat¬ 
ed  using  a  form  of  relaxation  labelling  [Zucker77]  with  dynamic  neighbourhoods  and 
compatibilities.  The  relationships  between  hypotheses  and  hypothesis  matching  pro¬ 
gress  are  the  important  factors  for  this  updating  scheme.  Using  the  best  hypothesis, 
predictions  are  determined  for  the  locations  of  the  objects  in  the  next  image  of  the  se¬ 
quence. 

Several  important  questions  can  be  raised  concerning  the  characteristics  of 
such  a  control  structure: 

1)  How  is  the  relaxation  labelling  procedure  adapted  to  a  time-varying  situation 
where  there  is  a  constant  stream  of  input  data  that  must  be  integrated  into  the  label 
certainty  factors? 

2)  How  is  noise  in  the  images  taken  care  of?  How  do  we  define  noise  in  time- 
varying  imagery? 

3)  How  is  the  temporal  segmentation  problem  solved,  i.e.,  when  does  ofie  motion 
end  and  the  next  one  begin  for  the  same  object? 

4)  How  are  the  changes  in  object  location  between  two  images  combined  into 
descriptions  that  extend  over  several  images? 

5)  Does  this  mechanism  truly  reduce  the  problem  of  shifting  foci? 

Answers  to  the  these  questions  will  be  given  in  this  document.  For  the  time 
being,  let  us  define  a  few  terms  that  will  be  needed  for  these  tasks.  Static  noise  is 
the  term  used  to  denote  the  degree  of  uncertainty  in  the  precise  intensity  value  of  a 
particular  point  in  an  image.  If  the  image  has  a  large  amount  of  static  noise,  then 
there  is  a  high  uncertainty  in  the  characteristics  of  a  particular  picture  element,  and 
therefore,  there  is  a  high  uncertainty  in  the  data  produced  by  the  image  analysis  com¬ 
ponent  in  a  computer  vision  system  in  terms  of  relationships  between  intensity  values 
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(for  example,  gradients).  Temporal  noise  is  the  analog  of  static  noise  in  time.  It 
describes  the  degree  of  uncertainty  of  a  particular  event  occurring  at  a  specific  time 
instant,  i.e.,  the  characteristics  of  the  time  instant  in  terms  of  which  events  are 
present  or  absent.  In  our  formulation,  a  large  of  amount  of  temporal  noise  will  mean 
that  there  is  a  correspondingly  large  amount  of  uncertainty  in  the  nature  of  the 
changes  that  the  image  analysis  component  "observes"  at  a  particular  time  instant, 
i.e.,  the  presence  or  absence  of  change  or  the  amount  of  change. 

Inter-image  description  is  the  term  used  to  denote  the  set  of  changes  that 
occur  between  a  pair  of  consecutive  images  in  a  sequence.  A  sequence  is  a  collection 
of  images  such  that  one  follows  the  other  in  time,  and  they  are  all  separated  by  a  con¬ 
stant  inter-image  time  interval.  Therefore,  one  can  number  them  as  the  first,  second, 
third,  etc.  Such  changes  are  usually  stated  in  terms  of  a  set  of  primitive  change 
types.  (The  term  "inter-frame  description"  is  used  in  [Nagel7B],  Our  definition  is  the 
same,  however,  we  reserve  the  term  "frame"  for  exclusive  use  as  a  knowledge 
representation  unit  in  order  to  avoid  confusion.)  If  we  wish  to  claim  that  our  system 
"understands"  motion  concepts,  it  is  not  enough  to  just  determine  the  movements  or 
changes  that  occur  between  a  pair  of  consecutive  images.  In  many  cases  it  is  desirable 
to  use  a  single  descriptive  term  for  all  the  inter-image  descriptions  in  several  adjacent 
image  pairs.  Such  a  motion  description  would  abstract  a  summary  description  from 
the  multitude  of  inter-image  descriptions  generated  for  the  image  sequence.  As  a  sim¬ 
ple  example,  consider  the  sequence  of  images  that  show  a  person  walking.  If  only 
inter-image  descriptions  are  generated,  the  result  is  a  complicated  and  cumbersome 
set  of  terms  (each  joint  of  the  body  requires  many  descriptive  terms  for  each  pair  of 
images  in  the  sequence).  It  would  be  necessary  if  we  wish  to  claim  that  our  system  has 
"understood"  these  motions  to  summarize  them  as  a  "wralk"  motion.  Vve  wdll  use  the 

v 

term  sequence  spanning  descriptions  to  denote  such  motion  concepts. 

The  term  instantiate  will  be  used  frequently  in  this  document  and  thus 
should  also  be  defined.  A  hypothesis  is  instantiated  wrhen  the  system  decides  that  it 
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provides  the  best  description  for  the  motion  of  a  particular  object  (or  group  of  objects) 
during  a  particular  time  interval.  Here,  "best"  means  that  the  system  has  a  much 
higher  confidence  in  the  correspondence  of  the  motion  characteristics  that  this  hy¬ 
pothesis  defines  with  the  changes  observed  in  the  image  sequence,  than  in  the 
correspondences  of  any  other  of  the  competing  hypotheses.  When  a  hypothesis  is  in¬ 
stantiated,  a  token  is  created,  that  is  a  data  structure,  that  conforms  to  the  structure 
of  the  original  hypothesis,  and  that  has  all  the  components  (slots)  of  the  hypothesis 
"filled”  with  data  abstracted  from  the  image  sequence. 

The  history  of  our  research  can  be  traced  in  past  papers  [Tsotsos77a], 
[Tsotsos77b],  [Tsotsos78],  [Tsotsos79a],  [Tsotsos79b]. 

The  remainder  of  this  document  elaborates  on  the  form  and  content  of  the 
knowledge  base  and  on  the  details  of  the  control  structure.  Chapter  2  discusses  mo¬ 
tion  understanding  in  general,  and  using  concepts  gleaned  from  past  research  in  the 
area,  develops  a  paradigm  for  recognition  of  visual  motion  concepts  from  a  sequence  of 
images.  In  Chapter  3,  the  formalism  for  the  representation  of  knowledge  to  be  used  by 
the  KB  is  described  with  several  examples  to  illustrate  the  concepts.  General  motion 
concepts  along  with  the  important  semantics  needed  for  adequate  description  are 
given  in  Chapter  4.  This  chapter  al~c  includes  a  basic  motion  concept  knowledge  base. 
Chapter  5  deals  with  our  problem  domain.  The  difficulties  encountered  in  analysis  of 
LV  images  are  discussed.  Details  are  given  about  the  problem  domain  as  derived  from 
cardiologists  and  medical  literature  and  then  the  KB  defined  for  them  is  presented. 
Not  all  of  it  is  shown  because  it  is  very  large;  however,  the  normal  LV  and  one  abnor¬ 
mality  class  are  described.  The  recognition  model  is  described  in  Chapter  6.  Details 
about  how  images  are  analyzed  in  order  to  extract  the  objects  in  them,  how  motions 
are  detected  and  described  between  images,  how  hypotheses  about  what  motions  are 
being  exhibited  over  a  sequence  of  images  are  activated  and  maintained,  how  hy¬ 
potheses  are  compared  against  one  another  and  how  decisions  are  made  for  the  selec¬ 
tion  of  the  best  one(s)  are  presented.  Chapter  7  presents  examples  of  the  partial  im- 
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plementation  of  our  MUS.  The  system  is  called  ALVEN  (A  Left  Ventricular  Wall  Motion 
Analysis  System).  The  important  concepts  are  illustrated  using  results  obtained  from 
the  implementation  of  the  relevant  portions  of  the  recognition  control  structure.  Fi¬ 
nally,  Chapter  B  presents  our  conclusions  and  discusses  directions  for  further 
research. 
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CHAPTER  2 


MOTION  UNDERSTANDING  -  A  THEORETICAL  FRAMEWORK 


2.1  Important  Concepts  from  Existing  Motion  Analysis  Systems 

* 

Motion  analysis  is  a  very  young  research  area.  The  approaches  attempted 
thus  far  have  been  devised  for  the  solution  of  specific  problems.  Very  little  research 
has  been  devoted  to  the  full  scope  of  motion  understanding  (MU)  (as  in  single  image 
understanding)  -  the  abstraction  or  recognition  of  motion  concepts  when  the  motion 
exists  over  several  images  and  not  just  two  (inter-image  descriptions  vs.  sequence 
spanning  descriptions).  That  is,  how  does  a  computer  system  determine  wrhich  single 
motion  concept  (or  a  related  group)  best  describes  a  motion  which  is  exhibited  over 
many  images?  Two  exceptions  are  Badler  [Badler75]  and  Tsotsos  [Tsotsos76j.  Howev¬ 
er,  in  the  past,  problem-specific  attempts  have  told  us  much  about  wrhat  is  necessary 
for  MU  in  general.  Excellent  surveys  are  found  in  [Nagel78]  and  [Martin78],  wThile  the 
proceedings  of  the  first  motion  analysis  workshop  [Badler79]  exposed  many  previously 
unknown  research  efforts. 

The  problem  of  motion  understanding  can  be  broken  down  into  several  sub¬ 
problems: 

1)  computer  vision 

a)  image  segmentation  and  object  recognition 

V 

b)  object  description 

c)  change  detection 

d)  motion  tracking 

e)  inter-image  change  description 
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2)  representation  of  knowledge 

a)  general  temporal  concept  representation. 

b)  problem  domain  motion  concept  representation 

c)  recognition  biased  knowledge  organization 

3)  recognition  control  structure 

a)  integration  of  descriptive  and  visual  concepts 

b)  change  and  focus  of  attention  mechanisms 

c)  temporal  segmentation 

d)  disambiguation  due  to  object  occlusion 

e)  goodness  of  fit  measures 

f)  generation  of  low  level  guidance 

g)  scene  sampling  rate  considerations 

h)  artifactual  motion  handling,  i.e.,  "temporal  noise" 

i)  generation  of  sequence  spanning  descriptions 

In  the  main,  past  motion  research  has  dealt  with  the  problems  under  the  first 
heading  above  -  the  vision  aspects  of  motion.  Several  good  concepts  can  be  found  in 
this  past  work  and  will  be  used  when  appropriate. 

Much  effort  has  been  devoted  to  the  solution  of  the  motion  detection  problem, 
i.e.,  the  determination  of  when  an  object  has  moved.  Several  systems  use  cross- 
correlation  techniques,  frequency  domain  comparisons,  or  matching  of  portions  of  im¬ 
ages  in  order  to  derive  an  average  motion  vector  for  the  region.  Potter  [Potter77]  used 
a  cross-shaped  template,  derived  from  one  image,  and  searched  for  in  the  next,  in  ord¬ 
er  to  detect  whether  or  not  the  object  represented  by  the  template  had  moved  and  by 
how  much.  Also,  simple  subtraction  of  one  image  from  another  can  yield  motion  infor¬ 
mation.  However,  good  alignment  and  constancy  of  intensity  values  are  crucial.  Rosen- 
feld  [Rosenfeld79]  used  an  extension  of  the  relaxation  procedure  for  edge  detection  in 
single  images  [Zucker77].  The  process  is  extended  to  three  dimensions  with  the  third 
one  being  time,  under  the  assumption  that  the  motions  are  small  -  on  the  order  of  one 
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picture  element  (pixel)  per  time  unit.  The  relaxation  process  is  then  run  over  the  en¬ 
tire  image  sequence  in  order  to  determine  which  points  are  in  motion.  This  process  is 
quite  complex  and  depends  heavily  on  parallel  computations.  In  our  particular  prob¬ 
lem  domain,  left  ventricular  wall  motion  analysis,  Tasto  [Tasto74]  used  the  current  lo¬ 
cation  of  the  left  ventricular  outline  in  his  cinecardioangiograms  as  a  guide  for  finding 
it  in  the  next  image.  This  is  just  one  of  several  examples  of  tracking  methods  which 
base  predictions  for  an  image  on  the  location,  velocity  and  trajectory  history  of  the  ob¬ 
ject.  We  shall  make  use  of  this  concept  as  well. 

Once  motion  has  been  detected  further  analysis  requires  additional  mechan¬ 
isms.  The  sub-structures  of  an  image  can  be  described  using  various  techniques,  the 
most  prominent  one  being  the  binding  of  image  regions  to  prototypical  descriptions 
and  organizing  these  into  hierarchies.  Sub-structures  can  be  of  the  form  of  the  ele¬ 
ments  of  Marr’s  primal  sketch  [Marr76],  i.e.,  blobs,  edge  elements,  etc.,  or  aggrega¬ 
tions  of  these  into  regions  which  have  semantic  meaning  [Freuder?6]. 

The  sub-structures  of  one  image  must  then  be  compared  to  these  of  the  next 
image  in  a  sequence  in  order  to  derive  inter-image  descriptions  (a  variant  on  the  track¬ 
ing  problem)  .  Several  diverse  methods  have  been  applied  to  this  problem  from  digi¬ 
tal  filtering,  transformation  and  comparisons  in  the  frequency  domain,  to  techniques 
that  extract  descriptors  that  are  semantically  related  to  the  observed  object.  We  will 
concentrate  on  the  latter,  or  conceptual,  ones  only.  One  such  technique  is  to  compare 
one  sub-structure  with  all  sub-structures  in  the  next  image.  This  clearly  is  very  expen¬ 
sive  computationally.  Sub-structure  similarity  can  also  be  derived  by  computing  the 
weighted  sum  of  the  characteristic  differences  between  regions.  Each  of  the  weights  is 
multiplied  by  a  strength  value  related  to  the  problem  domain  which  reflects  the  im¬ 
portance  of  the  characteristic  to  the  domain  [Price77].  Motion  and  sub-structure  com¬ 
parisons  can  also  be  used  to  perform  image  segmentation.  This  technique  [Thomp- 
son79]  does  not  require  previously  defined  objects  in  the  image  and  uses  motion  and 
brightness  similarities  in  order  to  construct  regions  of  similar  characteristics  that 
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define  the  locations  of  the  moving  objects  in  the  image  sequence.  The  technique  also 
uses  predicted  object  displacements  as  an  aid  to  setting  up  a  relaxation  process  neigh¬ 
bourhood  through  time.  The  prediction  is  a  general  one  and  not  related  to  the  motion 
capabilities  of  each  specific  object  type.  It  is  useful  for  such  a  purpose  to  define  the 
knowledge  that  is  necessary  to  reduce  the  search  space.  This  knowledge  must  aid  in 
selection  of  sub-structures  to  use  for  comparison  between  images  and  to  aid  in  the  in¬ 
terpretation  of  the  results  of  the  comparison.  Such  knowledge  should  be  made  explicit 
and  should  be  provided  in  a  formal  manner. 

The  sub-structure  matches,  once  found,  are  then  used  to  derive  inter-image 
descriptions  or  a  "sequence  description"  [Nagel78].  Such  descriptions  would  abstract 
out  any  essential  variations  between  images.  In  past  research  efforts,  these  have  been 
lists  of  co-ordinate  pairs  of  tracked  objects  or  lists  of  objects  along  with  attached 
descriptors  like  area  or  arclength  [Tasto74],  [Smalling76].  We  regard  such  descrip¬ 
tions  as  the  lowest  level  of  the  description  process.  More  abstract  descriptions  would 
specify  changes  over  several  images  as  a  unit,  and  would  have  a  stronger  conceptual 
basis. 

One  method  of  providing  information  for  higher  levels  of  abstraction  is  the 
processing  of  "spatio-temporal"  information  [Flinchbaugh79].  This  work  is  an  exten¬ 
sion  of  Marr’s  representation  scheme  [Marr77]  into  the  time  domain.  They  use  a  physi¬ 
cal  constraint  which  they  call  "confluence"  to  group  together  elements  of  the  primal 
sketch  which  exhibit  similar  motion  properties  over  a  sequence  of  images.  The  types  of 
information  produced  for  motion  are  average  motion  vectors  for  the  centroid  of  a 
group  of  elements,  and  the  components  of  the  actual  motion  vector  for  each  element 
through  the  centroid  of  the  group  and  perpendicular  to  that  vector.  However,  not 
much  has  yet  been  done  on  exactly  how  these  elements  may  be  used  to  abstract  higher 
level  descriptions. 

At  a  higher  conceptual  level  yet  is  the  wrork  done  by  Badler  [Badler75].  He 
defines  motion  concepts  such  as  the  verbs  "bounce"  and  "walk"  and  directionals  such 
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as  "towards”  and  "across",  using  lower  level  location,  orientation  and  spatial  relation 
changes.  Sequences  of  such  changes  define  the  motion  concept  under  consideration. 
A  low  level  process  would  analyze  the  images  of  the  sequence  and  provide  locative  and 
spatial  relationships  for  the  objects  in  the  scene,  i.e.,  the  processing  is  entirely 
bottom-up.  His  definitions  for  directionals  are  used  in  our  work  basically  as  found  in 
[Badler75];  they  will  be  described  in  Chapter  4  in  more  detail. 

Work  described  in  [Tsotsos76]  concentrates  mainly  on  the  analysis  of  the  se¬ 
mantic  components  necessary  for  motion  description,  and  on  a  representational 
scheme  for  verbs  which  is  based  on  Miller’s  [Miller72]  motion  verb  analysis.  The 
representation  formalism  to  be  presented  in  this  thesis  has  very  much  the  same 
flavour  as  that  in  [Tsotsos76]. 

All  of  the  research  described  in  the  preceeding  paragraphs  is  important  to 
motion  analysis.  However,  one  important  characteristic  is  lacking.  The  approaches 
solve  isolated  portions  of  the  general  problem,  and  provide  techniques  that  are  not 
directly  adaptable  for  other  application  domains  where  motion  is  important.  The 
methods  used  do  not  analyze  image  sequences  all  the  way  from  object  recognition  to 
high  level  conceptual  motion  descriptions.  In  addition,  several  temporal  concepts  have 
been  all  but  ignored  by  past  work: 

•  recognition  of  motion  concepts  that  appear  over  several  images  rather  than  just 
two,  e.g.,  the  "walking"  example  of  the  first  chapter,  (sequence  spanning  descriptions); 

•  segmentation  in  time,  i.e.,  when  does  one  motion  end  and  the  following  one  be¬ 
gin; 

•  the  form  of  a  general  motion  concept  knowledge  base,  in  terms  of  which  a  prob¬ 
lem  specific  knowledge  base  can  be  defined; 

•  how  is  a  sampling  rate  for  the  scene  chosen  so  that  all  significant  motions  will  be 
adequately  represented  in  the  image  sequence; 

•  what  can  be  done  to  reduce  the  effects  of  noise  in  the  images;  and, 

•  the  design  of  an  "attentive"  control  structure,  that  is,  one  that  determines  which 
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objects  to  track,  which  motions  are  related  to  one  another,  etc. 

This  thesis  attempts  to  provide  a  uniform  methodology  for  tackling  these 
problems. 


2.2  Intuition  and  Motivation 

Many  systems  which  do  single  image  understanding  use  a  common  principle 
to  aid  in  the  process:  guidance  from  a  high  level  component  to  aid  in  the  recognition  of 
ambiguous  objects  at  the  picture  level  [Freuder76],  [Tenenbaum77],  [Levine77],  [Kel- 
ly7l],  [Mackworth77],  [Ballard78].  We  can  call  this  principle  spatial  feedback:  name¬ 
ly,  the  abstractions  made  by  the  higher  levels  of  the  system  from  data  determined  by 
the  lower  levels,  are  used,  in  turn,  to  further  refine  the  analysis  done  by  the  lower  lev¬ 
els.  The  feedback  is  spatial  because  the  constraints  imposed  on  the  lower  levels  by  the 
high  levels  are  spatial  relations  or  physical  properties  which  must  be  satisfied  by  the 
objects  hypothesized  to  be  in  the  image. 

This  same  principle  can  be  extended  into  the  time  domain  -  guidance  based 
on  temporal  feedback,  for  use  in  the  analysis  of  image  sequences.  The  following 
paragraphs  give  an  intuitive  view  of  how  this  extension  can  be  viewed. 

For  recognition  in  arbitrary  single  images,  there  are  no  clear  a  priori  expec¬ 
tations  as  to  where  to  look  for  objects  -  the  entire  image  must  be  searched,  with  pro¬ 
cessing  for  each  image  proceeding  in  the  same  manner.  However  if  motion  is  con¬ 
sidered,  the  processing  scheme  can  be  altered.  Again,  for  the  first  image,  the  entire 
process  must  take  place  without  expectations.  This  first  image  determines  the  objects, 
their  locations,  their  orientations,  and  their  spatial  relations.  Such  information  alone 
can  be  of  some  help  to  the  1owt  level  recognizer  for  successive  images.  In  addition,  a 
basic  premise  useful  for  motion  understanding  is  that  the  motion  is  continuous.  Using 
this  premise,  the  low  level  recognizer  knowrs  that  for  any  one  object,  its  location  in  the 
next  image  will  be  "close"  to  its  current  location.  Here  "close"  means  that  all  direc¬ 
tions  of  motion  are  equi-probable  and  that  there  is  some  uniform  limit  imposed  on  all 
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objects  restricting  the  maximum  displacement  in  2D  in  the  time  interval  between  two 
images.  In  this  way  the  low  level  recognizer  can  search  limited  areas  of  the  image  to 
find  the  objects  recognized  in  the  previous  image. 

It  should  be  pointed  out  here  that  it  would  be  very  difficult  for  a  system  using 
limited  spatial  feedback  to  analyze  the  motions  that  the  objects  exhibit.  The  reason 
for  this  is  that  motion  is  a  temporal  process.  Therefore,  there  must  be  a  mechanism 
which  can  relate  or  combine  the  changes  of  locations  of  objects  in  successive  images. 
For  a  system  with  only  spatial  feedback,  no  expectations  can  be  produced  for  the  mo¬ 
tions  in  successive  images.  This  means  that  the  low  level  must  determine  all  the  data, 
and  this  entails  determination  of  locations  and  orientations  for  each  object  and  all  spa¬ 
tial  relations  between  all  objects  in  each  image  regardless  of  whether  they  are  needed 
or  not.  Motion  expectations  are  necessarily  determined  by  combining  current  ob¬ 
served  motions  with  expected  future  ones  and  this  requires  temporal  knowledge. 

Consider  such  a  spatial  feedback  system  for  a  moment.  It  is  clear  that  all  re¬ 
lations  between  all  objects  are  not  necessary  -  particularly  for  those  objects  which  do 
not  move.  In  fact,  the  motions  of  most  objects  would  not  be  related  to  other  objects  at 
all.  In  addition,  from  some  simple  knowledge  about  the  motion  capabilities  of  the  ob¬ 
jects,  it  is  possible  to  cut  down  the  search  space  for  each  object  which  the  low  level 
must  consider.  For  example,  consider  a  first  image  which  shows  a  automobile  on  a 
road.  The  automobile  and  road  and  other  objects  have  already  been  recognized  and 
described  by  another  process.  Cars  do  not  usually  move  up  into  the  air,  down  into  the 
street  or  sideways  under  their  own  power  (let  us  ignore  stunt  cars  or  strange  accidents 
for  the  moment  and  consider  only  normal  driving  habits).  They  move  only  in  either  a 
forward  or  backward  direction.  Thus,  for  the  second  image  the  low  level  recognizer 
must  search  only  ahead  of  or  behind  the  location  of  the  car  in  [ he  first  image,  and  not 
above  or  below  and  at  all  directions  in  between.  As  well,  when  cars  move,  they  cannot 
change  direction  (i.e.,  between  the  forward  and  backward  direction)  without  first  stop¬ 
ping.  Thus,  the  search  can  be  restricted  to  only  one  direction  -  the  current  direction 
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of  motion.  In  addition,  there  are  rough  guidelines  for  the  maximum  acceleration  at 
various  speeds  and  also  the  physical  fact  that  the  speed  function  is  continuous.  This  al¬ 
lows  determination  of  the  range  of  possible  locations  of  the  car  for  the  known  time  in¬ 
terval. 

It  should  be  clear  by  now  that  some  additional  component  is  necessary  to  the 
motion  understanding  system  described  thus  far,  i.e.,  one  which  interacts  with  the  low 
level  recognizer  during  processing  and  one  which  knows  something  about  the  objects 
observed  in  the  images.  This  additional  component  is  a  higher  level  guidance  process 
which  utilizes  a  knowledge  base.  The  guidance  process  must  consist  of  many  integrat¬ 
ed  components  and  will  be  described  in  a  later  chapter. 

2.3  Our  Motion  Understanding  Frame-work 

All  of  the  problems  described  in  section  one  of  this  chapter  -  change  detec¬ 
tion,  tracking,  inter-image  descriptions  -  are  necessary  for  the  general  task  of  motion 
understanding.  In  addition  there  are  the  problems  of  representation  and  use  of  gen¬ 
eral  and  problem-specific  knowledge,  and  of  recognition  of  temporal  concepts  not  only 
between  pairs  of  images,  but  of  motions  which  extend  over  severed  images  (sequence 
spanning  descriptions).  All  of  these  tasks  must  be  integrated  into  a  single  recognition 
methodolgy  which  solves  all  of  them  in  a  conceptually  clean  and  satisfying  manner. 

In  order  to  do  this,  the  choice  of  representation  formalism  and  of  recognition 
control  structure  design  is  crucial.  A  brief  review  of  two  important  large  application 
systems  is  therefore  appropriate  here. 

Perhaps  the  most  prominent  and  important  recognition  paradigm  in  artificial 
intelligence  is  "hypothesize-and-test".  Two  large  systems  which  use  this  paradigm  are 
PIP  [Szolovits78]  and  HEARS AY-II  [Lesser77],  [Hayes-Roth77]. 

PIP’s  domain  is  the  diagnosis  of  kidney  diseases.  PIP  has  a  representation 
for  its  domain  knowledge  which  is  frame-based:  knowledge  units  have  prerequisite 
findings  (triggers)  which  determine  their  appropriateness,  neccessary  and  sufficient 
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findings,  links  to  causally  related  complementary  knowledge  units  and  links  to  compet¬ 
ing  units  for  differential  diagnosis.  Patient  findings  are  used  to  determine  which 
knowledge  units  should  be  activated  as  hypotheses.  Subsequent  findings  (either  given 
the  system,  or  asked  for  by  the  system)  are  used  to  determine  goodness-of-fit  of  the 
hypothesis.  The  links  to  complementary  hypotheses  are  used  to  determine  further 
support  for  the  hypothesis,  either  positive  or  negative,  while  competing  hypotheses  are 
used  to  determine  alternatives  if  the  hypothesis  proves  faulty.  Each  hypothesis  has  as¬ 
sociated  with  it  a  "strength”  value  which  is  a  measure  of  its  goodness  of  match.  Initial 
values  of  strength  are  known  for  each  hypothesis  a  priori  and  a  mechanism  is  provided 
for  propagation  to  other  related  hypotheses.  This  system,  although  successful  to  a 
high  degree,  faces  a  "focus"  problem  -  that  is,  the  focus  of  attention  shifts  depending 
on  the  particular  order  of  findings  that  are  input. 

HEARSAY-II  uses  the  hypothesize-and-test  paradigm  in  quite  a  different  way 
in  the  domain  of  speech  recognition.  Knowledge  units  are  still  activated  by  prere¬ 
quisites,  and  this  activation  process  is  data-directed  as  in  PIP.  The  focus  of  attention 
mechanism  is  more  sophisticated  however,  in  that  there  are  several  heuristics  used  to 
guide  it.  Hypotheses  are  considered  depending  on  specific  criteria  :  highest  strength 
first,  significance  of  possible  effect,  efficiency,  and  goal  satisfaction.  In  addition,  if  a 
hypothesis’  certainty  factor  fails  below  a  threshold,  which  is  dynamic  depending  on  the 
current  best  hypothesis,  the  hypothesis  is  put  aside  temporarily.  This  presents  the 
danger  of  getting  trapped  on  false  peaks.  Other  problems  arise  because  hypotheses 
can  be  shared  among  contexts  and  there  is  no  mechanism  for  determining  similarity  of 
contexts.  Conclusions  reached  through  experience  with  the  HEARSAY-II  system  in¬ 
clude: 

•  hypothesis-and-test  is  computationally  feasible  for  solution  of  knowledge  inten¬ 
sive  tasks; 

•  there  is  a  need  for  high  selectivity  in  hypothesis  certainty  factors  in  order  to  be 
able  to  choose  from  among  many; 
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•  a  good  focus  mechanism  needs  to  have  a  global  view  of  the  problem,  and  data- 
directed  control  allows  for  quick  re-focussing. 

We  will  see  how  our  methodology,  which  is  also  based  on  the  hypothesize- 
and-test  paradigm,  handles  the  problems  encountered  by  these  two  systems. 

An  approach  put  forward  by  Zucker  [Zucker78]  seemed  to  us  to  have  the 
promise  of  dealing  with  the  kinds  of  problems  mentioned  above.  It  combines  the  feed¬ 
back  techniques  used  in  edge  and  line  detection  [Zucker77]  in  images  with  the 
hypothesize-and-test  paradigms  discussed  above.  The  knowledge  units  used  are  "rules" 
-  these  rules  however,  are  quite  similar  in  content  to  the  frames  described  above  and 
vice  versa.  Therefore,  we  will  simply  call  them  knowledge  units.  Their  characteristics 
are:  each  knowledge  unit  has  a  set  of  prerequisites  which  determines  its  applicability, 
the  satisfaction  of  which  is  used  to  activate  the  unit  as  a  hypothesis;  each  hypothesis 
has  a  certainty  factor  associated  with  it  during  the  recognition  process;  and  there  is  a 
conclusion  which  is  added  to  the  context  of  recognition  if  a  knowledge  unit  is  satisfied. 
In  addition,  there  is  a  mechanism  for  relating  one  knowledge  unit  to  another. 

The  problems  encountered  by  PIP  and  HEARSAY,  and  other  similar  systems, 
are  due  to  inexact  information  -  either  incorrect  data,  extraneous  information  (part  of 
the  problem  domain  but  not  directly  related  to  the  task  at  hand),  or  noisy  data.  This 
causes  mis-matching  of  hypotheses,  leading  to  erroneous  partial  matches  and  multiple 
matches  where  in  reality  only  one  hypothesis  should  match.  Consider  these  problems 
in  the  simpler  context  of  edge  detection  in  single  images.  Line  quantization,  lighting 
and  shading  effects  cause  erroneous  input,  while  noise  adds  another  dimension  of  error 
to  the  process.  With  feedback,  as  in  relaxation  labeling  [Zucker77],  proper  functioning 
under  such  conditions  may  be  possible.  One  reason  for  this  is  that  the  effects  of  isolat¬ 
ed  instances  of  stimuli  are  minimized.  For  example,  in  static  situations,  such  as  in  a 
single  image,  a  noise  point  is  not  related  in  a  meaningful  way  to  its  neighbours  in  the 
image.  Similarily,  in  dynamic  situations,  where  events  change  with  time,  spurious 
events  are  similarly  unrelated  to  the  events  in  a  sequence  or  context,  i.e.,  its  neigh- 
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bours  in  time.  The  relaxation  procedure,  by  using  slowly  responding  feedback,  can 
reduce  the  effect  of  such  erroneous  situations. 

Response  of  a  focus  mechanism  based  on  feedback  is  not  instantaneous  - 
there  is  an  inherent  delay.  Past  systems,  such  as  PIP  and  KEARSAY-II,  have  attempted 
to  provide  a  completely  updated  view  of  the  analysis  of  the  problem  domain  at  each  in¬ 
stant  during  processing.  Therefore,  any  errors  in  the  input  data  would  be  completely 
integrated  as  if  they  were  valid  data,  rather  than  waiting  to  see  if  the  trend  is  indeed 
that  which  is  suggested  by  the  data.  The  focus  of  the  system  should  not  shift  abruptly  - 
it  should  only  change  if  the  stimulus  dictating  the  change  is  present  over  a  significant 
period  of  time.  Such  time  periods  are  clearly  problem  dependent. 

In  motion  understanding,  such  a  concept  is  vital.  Sequences  of  images 
present  difficult  problems:  occlusion  of  objects,  so  that  locations  may  sometimes  only 
be  guessed  at;  hypotheses  with  overlapping  expected  time  intervals  which  describe  the 
possible  motion  concepts  of  the  domain;  and,  particularly  for  our  domain  of  left  ven¬ 
tricular  wall  motion,  poor  quality  images,  either  due  to  poor  clinical  technique  or  to 
poor  imaging  technique.  Feedback  principles  allow  us  to  tackle  these  problems.  One 
way  in  which  isolated  stimuli  such  as  noise  can  be  handled  is  by  providing  descriptions 
in  the  KB  at  varying  levels  of  abstraction,  from  coarse  to  detailed.  Feedback  can  then 
be  present  between  these  levels,  with  the  coarse  description  placing  strong,  more  glo¬ 
bal  constraints  on  the  detailed  description,  thus  assisting  in  the  removal  of  isolated  er¬ 
ror  stimuli.  In  addition,  since  the  relaxation  process  accumulates  the  evidence  of  the 
matching  history  of  a  hypothesis,  it  is  very  difficult  for  an  isolated  error  event  to 
negate  a  large  positive  history  of  successful  matching  (or,  for  that  matter,  a  history  of 
matching  failures). 

However,  the  proposal  by  Zucker  requires  some  refining.  Relaxation  labeling 
as  applied  to  images  uses  the  spatiai  adjacency  of  pixels  in  order  to  determine  which 
set  of  pixels  to  combine  and  test  for  validity.  The  updating  of  certainty  factors  for  the 
competing  hypotheses  for  that  pixel  is  then  based  on  these  comparisons  through  the 
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compatibility  factors  between  the  label  types.  There  is  no  similar  mechanism  for  more 
general  hypotheses  as  required,  for  example,  by  PIP.  Also,  there  are  no  particular 
change  or  focus  of  attention  mechanisms  mentioned.  This  is  because  the  edge  opera¬ 
tor  is  applied  to  the  image  only  once.  In  more  general  recognition  systems,  there  is  a 
steady  stream  of  input  data  from  the  problem  task.  This  necessitates  change  and 
focus  of  attention  mechanisms  to  aid  in  determination  of  which  data  are  salient.  We 
will  develop  such  mechanisms  in  this  thesis. 

Let  us  get  to  specifics  now  and  show  how  the  feedback  mechanism  in  [Zuck- 
er78]  can  be  adapted  to  solve  the  motion  understanding  problem.  In  order  to  get  a 
better  grasp  of  feedback  as  defined  in  control  theory  (from  which  Zucker  derived  his 
inspiration),  it  is  useful  to  define  the  components  of  a  feedback  system  in  more 
abstract  terms  than  those  usually  found  in  an  engineering  text. 

Consider  the  configuration  of  a  standard  control  system  as  shown  in  Figure 
2.1.  Note  that  the  system  really  consists  of  five  mechanisms: 

1)  a  mechanism  to  generate  the  system’s  output  in  the  terms  of  the  outside  world 
("process”)  -  the  controlling  signal; 

2)  a  mechanism  to  provide  input  from  the  outside  world  with  respect  to  the  out¬ 
put  in  terms  the  system  can  understand  ("feedback  transducer”)  -  a  measure  of  the 
effect  of  the  controlling  signal; 

3)  a  mechanism  to  detect  whether  or  not  there  is  a  difference  between  the  data 
from  the  outside  world  and  the  system’s  desired  state  of  the  outside  world  ("summa¬ 
tion  of  signals")  ; 

4)  a  mechanism  to  determine  the  compensatory  action  if  differences  are  detect¬ 
ed  ("controller");  and,  finally, 

5)  a  mechanism  to  generate  a  reference  signal  which  represents  the  system’s 
desired  state  for  the  outside  world  ("reference”). 

In  such  a  process,  any  deviation  from  the  output  required  by  the  system  (the 
reference)  due  to  noise  or  errors,  is  detected  and  causes  a  transient  response  until  a 
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FIGURE  2.2  The  Knowledge  Based  Feedback  Model  for  Analysis  of 

Time-Varying  Imagery 
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new  stable  state  is  achieved.  The  transient  response  represents  the  change  of  state  of 
the  system  and  is  guided  by  the  feedback  transducer.  The  feedback  signal  is  com¬ 
pared  to  the  reference  signal  and  their  difference  determines  the  amount  of  change 
the  outside  world  must  undergo.  This  change  is  constantly  monitored  by  the  feedback 
transducer  and  adjusted  as  necessary.  In  particular,  feedback  may  be  "constructive’', 
in  which  case  it  helps  the  system  dampen  or  stabilize  its  transient  output,  or  "destruc¬ 
tive",  in  which  case  it  leads  the  system  away  from  its  current  output.  The  system  is 
"unstable"  if  only  destructive  feedback  is  present. 

However,  the  system  in  Fig.  2. 1  is  a  control  system,  and  not  a  recognition  sys¬ 
tem.  In  a  control  system,  the  desired  final  state  is  known  a  priori,  while  in  the  recogni¬ 
tion  case,  this  final  state  is  unknown  and  must  be  determined  by  the  system.  There¬ 
fore,  several  modifications  to  this  configuration  are  necessary  if  we  wish  to  apply  it  to 
the  task  of  motion  understanding.  Let  us  first  re-draw  the  diagram  of  Figure  2.1  re¬ 
defining  some  of  the  components  slightly  -  see  Figure  2.2.  The  "process"  becomes  the 
"expectation  generation"  -  it  provides  information  to  the  feedback  transducer  about 
what  the  system  expects  the  outside  world  to  be  doing.  The  "feedback  transducer" 
remains  the  same  -  it  provides  data  about  the  outside  world  to  the  system  in  compati¬ 
ble  terms.  "Matching"  replaces  the  summation  of  signals,  because  a  more  general 
function  is  required  than  simply  adding  together  two  signals  in  order  to  detect 
differences.  The  reference  is  unknown  in  a  recognition  task  (as  opposed  to  a  control 
task)  and  thus  the  controller-reference  combination  is  replaced  by  a  knowledge  base  of 
general  and  problem-specific  knowledge,  a  set  of  currently  active  hypotheses,  (togeth¬ 
er  they  form  the  "reference"  in  the  figure),  and  a  process  which  maintains  this  set  of 
hypotheses  (the  "reference  controller"). 

Motion  detection  and  tracking  are  encompassed  within  the  expectation  gen¬ 
eration  and  feedback  components,  while  inter-image  descriptions  are  done  within  the 
feedback  transducer.  The  problems  of  being  trapped  on  false  peaks  as  in  HEARSAY,  or 
of  shifting  foci  as  in  PIP  should  be  reduced  with  our  control  structure.  Many  false 
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peaks  should  not  even  become  peaks  because  of  the  inherent  delay  in  the  feedback 
response  to  stimuli.  Furthermore,  the  focus  should  not  shift  quickly  because  of  exactly 
the  same  reason.  The  determination  of  contexts  which  share  hypotheses  or  the  deter¬ 
mination  of  similarity  of  hypotheses  is  handled  by  explicit  features  of  our  representa¬ 
tion.  The  representation  will  be  described  in  detail  in  Chapter  3. 

The  representational  formalism  to  be  used  for  motion  concepts  is  as  impor¬ 
tant  as  the  recognition  control  structure  because  the  control  structure  is  driven  by  the 
representational  features  of  the  knowledge  base.  Each  motion  concept  has  a  frame 
associated  with  it  that  defines  it.  A  frame  is  definable  by  the  user  and  may  encom¬ 
pass  as  large  a  knowledge  unit  as  desired.  Frames  have  as  many  slots  as  necessary 

that  form  its  parts,  thus  defining  a  PART _ OF  hierarchy  of  description.  Each  level 

down  in  this  hierarchy  provides  a  more  detailed  form  of  description  for  the  motion  con¬ 
cept.  Slots  come  in  two  varieties:  prerequisite  and  dependent.  Prerequisite  slots 
specify  concepts  which  must  be  observed  before  the  frame  can  be  instantiated,  while 
dependents  provide  additional  semantic  components  that  are  included  along  with  the 
frame  concept  on  instantiation.  In  addition,  slots  can  have  constraints  associated  with 
them.  Frames  are  organized  along  the  I S__A  relationship  in  order  to  relate  general 
and  specific  motion  frames.  Similarity  links  [Minsky75]  relate  one  frame  to  another 
and  aid  in  altering  the  system's  active  hypothesis  set.  Using  such  a  formalism,  a 
knowledge  base  is  defined  for  the  motion  concepts  necessary  for  the  application 
domain. 

Let  us  summarize  our  motion  understanding  framework.  Using  a  KB  as 
described  above,  processing  proceeds  image  by  image.  Our  system  looks  at  one  image 
at  a  time,  and  once  it  is  finished,  the  image  is  discarded.  A  requirement  is  that  the  ob¬ 
jects  in  the  first  image  in  the  sequence  be  identified  and  classified  prior  to  motion 
analysis.  Such  a  process  may  be  accomplished  manually  or  by  techniques  such  as  in 
[Tenenbaum77],  Expectations  produced  b}'  the  system  guide  the  low  level  process  in 
identifying  objects  in  subsequent  images.  This  precludes  new  objects  from  entering  the 
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scene.  If  an  object  is  not  found  at  precisely  the  same  location  in  the  image  as  predict¬ 
ed  by  the  high  levels  of  the  system,  then  the  object  has  moved  and  presumably,  motion 
has  been  detected.  This  motion  is  one  of  two  forms:  either  the  object  is  being  tracked, 
or  the  object  has  just  started  or  stopped  moving.  If  the  object  is  being  tracked,  the 
predictions  specify  a  range  of  possible  locations  for  it  in  the  next  image.  The  object 
descriptions  are  in  terms  of  the  low  level  vision  constructs,  such  as  edges  or  regions  in 
the  image,  along  with  descriptors  such  as  type,  axes,  area  and  arclength.  This  descrip¬ 
tion  is  called  the  essential  trace  for  the  object. 

Pairs  of  time-wise  adjacent  essential  traces  are  combined  into  the  essential 
kineses  of  each  object.  Essential  kineses  are  defined  in  terms  of  location  changes  of 
points,  length  changes  of  axes  and  perimeters,  area  changes  and  shape  changes.  These 
four  primitive  kineses  provide  an  intermediate  representation  for  relating  quantitative 
changes  to  qualitative  ones.  The  kineses  are  used  to  match  against  the  hypotheses 
which  are  active  for  a  particular  object's  motion. 

The  system  must  start  with  an  initial  hypothesis  for  the  motion  of  each  object 
in  the  image  sequence.  This  initial  "guess"  need  not  be  a  perfect  one:  however,  it  is  as¬ 
sumed  that  the  knowledge  base's  hypotheses  are  properly  related  to  one  another  and 
that  it  is  complete  in  this  respect.  The  essential  kineses  are  described  in  the  same 
terms  as  the  lowest  level  of  motion  description  for  each  hypothesis  of  the  knowledge 
base.  It  may  be  necessary  in  some  cases  (when  the  motions  are  only  very  coarsely 
described)  to  abstract  higher  level  descriptive  terms  from  the  kineses.  Matching 
failures  between  expected  and  actual  kineses  are  used  by  the  change  of  attention 
mechanism.  These  failures  are  represented  in  terms  of  exception  frames  which  con¬ 
tain  any  information  necessary  to  the  change  of  attention  process  so  that  proper  selec¬ 
tion  can  be  made  of  alternate  hypotheses.  This  selection  is  made  via  the  similarity 
links  which  are  present  in  the  hypotheses.  Each  active  hypothesis  is  related  to  the  oth¬ 
er  active  ones  by  its  conceptual  adjacency.  Conceptual  adjacency  is  denned  in 
terms  of  the  following  primitive  relationships  of  the  knowledge  base  :  1S_A,  PART_OF, 
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similarity  and  time  course. 


The  focus  of  attention  mechanism  ranks  the  hypotheses  on  the  basis  of  their 
updated  certainty  factors  in  order  to  determine  the  best  ones.  Each  hypothesis,  when 
activated,  receives  an  initial  certainty  factor  equal  to  that  of  the  hypothesis  that  ac¬ 
tivates  it.  Relaxation  labelling  is  then  used  to  update  the  certainty  factors.  The  relaxa¬ 
tion  process  is  based  on  conceptual  adjacency  that  specifies  which  hypotheses  are 
competitors  and  which  ones  are  complementary  and  in  what  respect.  The  best  hy¬ 
potheses  (highest  ranked)  then  are  used  to  derive  the  expectations  for  the  next  image. 

The  instantiation  of  a  hypothesis  poses  a  unique  problem:  the  temporal  seg¬ 
mentation  of  time-wise  adjacent  events.  In  other  words,  where  does  one  motion  start 
and  the  previous  one  end  for  a  particular  object?  This  problem  is  solved  by  analysis  of 
the  time  course  of  certainty  factors  for  the  two  motions  involved. 

The  following  chapters  will  discuss  the  framework  described  above,  com¬ 
ponent  by  component,  in  detail. 
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CHAPTER  3 


REPRESENTATION  OF  MOTION  CONCEPTS 

3.1  What  Needs  to  be  Represented? 

The  main  concern  of  this  representational  formalism  is  how  motion  concepts 
are  to  be  represented  and  organized  into  a  knowledge  base  (KB).  The  intent  is  to  pro¬ 
vide  a  formalism  for  representing  general  motion  concepts  such  that  problem-specific 
ones  can  be  represented  in  terms  of  the  general  concepts.  Furthermore,  the  form  of 
the  KB  must  be  such  that  it  can  be  used  by  an  appropriate  control  structure  to  guide 
recognition  of  the  problem-specific  concepts  stored  in  it.  The  limitations  of  our  formal¬ 
ism  were  outlined  in  Chapter  1. 

There  are  several  types  of  motion  concepts: 

•  speed  of  an  object,  e.g.,  30km/hr,  or  slowly: 

•  direction  of  motion,  e.g.,  35  degrees  from  the  horizontal  axis; 

•  descriptive  motion  verbs,  e.g.,  pull,  rotate,  stretch; 

•  qualitative  trajectory  descriptions  -  directionals,  e.g.,  towards,  upwards; 

•changes  of  an  object's  properties,  e.g.,  length,  area,  shape  changes; 

•  temporal  concepts:  duration,  start  and  end  time  of  an  event,  relative  timing  of 
events,  e.g.,  event  A  occurs  before  event  B. 

In  addition  to  providing  a  coherent  scheme  for  representing  such  concepts, 
the  formalism  must  take  several  other  constraints  into  account: 

•  The  data  involved  often  have  a  "fuzzy”  quality.  There  are  ranges  of  acceptable 
values  to  contend  with,  as  opposed  to  single  correct  values,  as  well  as  overlapping 
ranges  of  acceptable  values  between  conflicting  concepts.  For  example,  event  A  may 
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have  an  acceptable  end  time  of  between  8  to  12  time  units  from  the  start  of  the  event, 
while  event  B  may  have  an  acceptable  start  time  of  between  10  and  14  time  units  from 
the  start  of  event  A.  These  are  overlapping  ranges  of  acceptable  values. 

•  The  representation  must  have  a  strong  conceptual  basis  with  qualitative  con¬ 
cepts  related  to  quantitative  ones.  This  is  because  the  knowledge  base  consists,  in  part, 
of  verbally-specified  knowledge,  and  also  because  it  is  to  be  used  for  question¬ 
answering  about  its  contents  and  the  analysis  performed. 

•  Instantiated  data  and  generic  data  must  contain  sufficient  information  for  the 
approximate  re-generation  of  the  analysed  film  or  of  generic  classes  of  motion.  In  oth¬ 
er  words,  the  descriptions  must  be  complete  enough  to  be  converted  back  to  image  se¬ 
quences,  such  that  a  cardiologist  could  recognize  the  motions  as  belonging  to  the  class 
specified. 

•  The  formalism  must  be  capable  of  representing  concepts  at  different  levels  of 
abstraction,  depending  on  the  level  of  expertise  of  the  person  who  enters  the 
knowledge.  For  example,  consider  two  of  the  ways  of  specifying  the  motions  of  a  seg¬ 
ment  of  the  left  ventricle:  either  as  "towards  the  centroid  with  the  same  speed  as  its 
neighbouring  segment"  or,  with  much  more  detail,  "the  segment  moves  at  a  speed  of 
3cm./sec.  with  a  trajectory  of  47  degrees  away  from  the  horizontal  axis".  Both  types 
of  descriptions  must  be  handled  because  there  are  cases  where  only  coarse  descrip¬ 
tions  are  available  that  must  be  integrated  with  more  detailed  descriptions. 

3.2  Representational  Building  Blocks 

The  representation  of  knowledge  used  here  has  its  roots  in  semantic  network 
representations  [Brachman79],  and  in  particular,  the  PSN  formalism  [Levesque79]. 
The  syntax  that  is  used  in  the  example  frames  is  summarized  in  Appendix  A. 

3.2. 1  Frame  Definition 

The  basic  entities  of  the  representation  are  called  frames  and  are  used  to 
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model  abstract  motion  concepts  such  as  that  of  CYCLE  or  WALK,  and  abstract  object 

concepts  such  as  HEART  or  PERSON.  The  instantiations  of  the  abstract  concepts  of  the 

real  world  are  modelled  in  the  representation  by  (frame)  tokens.  The  instance_of 

relationship  relates  tokens  to  the  frames  of  which  they  are  instances.  Thus  the  token 

"john",  which  represents  a  particular  person,  is  related  to  PERSON  through  the 

instance_of  relationship.  The  statement 

frame  PERSON 
end 

simply  declares  (defines)  the  frame  PERSON,  while 

token  john  instance_of  PERSON 
end 

defines  the  token  named  "john”  as  being  an  instance  of  PERSON.  Every  token  must  be 
an  instance  of  at  least  one  frame. 

Each  frame  may  have  an  associated  description  of  its  internal  structure  that 
defines  the  semantic  components  of  the  concept  that  the  frame  represents.  For  mo¬ 
tion  concepts,  the  internal  structure  of  a  frame  defines  the  semantic  components  of 
the  motion,  such  as  speed,  trajectory  or  duration.  On  the  other  hand,  object  concept 
frames  have  an  internal  structure  that  defines  their  physical  parts  and  the  properties 
posessed  by  them.  The  internal  structure  of  PERSON,  for  example,  may  include  a  right 
leg,  a  left  leg,  a  right  arm,  a  left  arm,  a  torso,  a  neck  and  a  head.  The  internal  struc¬ 
ture  of  a  frame  is  defined  by  slots,  each  of  which  has  a  name  and  an  associated  type. 
The  type  relates  the  slot  to  other  frames  that  specify  the  kind  of  tokens  that  can  fill 
(be  bound  to)  it.  For  example,  the  definition  of  PERSON  may  be  extended  to  the  follow¬ 
ing  example,  in  order  to  indicate  the  physical  parts  of  PERSON  through  seven  slots: 

frame  PERSON  with 

right__leg  :  RIGHT_LEG  ; 
left_leg  :  LEFT_LEG  ; 
neck  :  NECK  ; 
head  :  HEAD  ; 
right_arm  :  RJGHT_jVRM  ; 
left_arm  :  LEFT_ARM  ; 
torso  :  TORSO  ; 

end 

Note  that  identifiers  in  capital  letters  are  used  to  denote  frames  while  those  in  lower 
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case  letters  are  used  to  denote  slot  names  and  tokens.  The  lower  case  bold  letters  in¬ 
dicate  formalism  keywords.  Thus, 
left_leg  :  LEFT_LEG  ; 

defines  a  slot  with  name  left_leg  and  type  LEFT_LEG.  LEFT_LEG  is  another  frame,  and 
is  defined  separately  in  the  knowledge  base. 

Instances  of  a  frame  must  have  an  internal  structure  which  conforms  to  that 
of  the  frame,  e.g., 

t oken  john  instance _ of  PERSON  with 

right_leg  :  right_leg_l  ; 
left_leg  :  left_leg_l  ; 
neck  :  neck_l  ; 

head  :  head _ 1  ; 

right_arm  :  right_arm_l  ; 

left_arm  :  left_arm _ 1  ; 

torso  :  torso__l  ; 

end 

where  right_leg_l,  left_leg_l,  torso„l,  neck_l,  head _ 1,  right_arm__l,  left_leg_l  are 

instances  of  RIGHT__LEG,  LEFT_LEG,  TORSO,  NECK,  HEAD,  RIGHT_ARM,  and  LEFT__ARM 
respectively. 

Since  the  knowledge  base  using  this  representation  is  to  be  used  to  guide  a 
recognition  process,  consideration  must  be  given  to  distinguishing  concepts  that  are  to 
be  observed  in  the  outside  world  from  those  that  are  to  be  computed  by  the  system  us¬ 
ing  data  from  the  outside  world.  There  is  a  real  difference  in  semantic  components 
here,  and  not  one  just  for  implementation  convenience.  As  we  will  describe  in  the  next 
chapter,  motion  concepts  have  an  associated  set  of  semantics  that  define  the  concept. 
Some  of  these  are  observable  or  can  be  described  in  terms  of  more  general  concepts, 
and  others  must  be  computed  from  observable  data.  The  computed  components  are 
those  that  specialize  the  motion  concept,  i.e.,  in  the  way  that,  for  example,  a  ''falling” 
event  is  a  more  specific  form  of  a  "translate”  event.  Therefore  slots  are  classified  into 
two  categories  depending  on  whether  they  are  functionally  dependent  on  other  slots. 
Prerequisite  slots  define  types  and  constraints  of  expected  motions.  Dependent 
slots,  on  the  other  hand,  are  assigned  values  that  depend  on  the  values  of  the  tokens 
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that  are  instances  of  the  prerequisite  slots.  For  example,  the  definition  of  PERSON  can 
be  modified  to  : 


frame  PERSON  urith 
prerequisites 

right_leg  :  RIGHT_LEG  ; 
left_leg  :  LEFT_LEG  ; 
right_arm  :  RIGHT__ARM  ; 

left_arm  :  LEFT _ ARM  ; 

neck  :  NECK  ; 
head  :  HEAD  ; 
torso  :  TORSO  ; 
dependents 

height  :  LENGTH_V  urith 

height  left_leg.  arclen  4-  torso. arclen  + 

neck.arclen  +  head. long_axis. arclen  ; 

end 

LENGTH_V  is  the  frame  for  length  values,  i.e,  it  is  a  data  type  with  dimension. 

There  are  several  such  data  types,  similar  to  LENGTK_V  -  AREA _ V,  SPEED_V,  TRAJ_V, 

ANGLE_V,  etc.  Also,  the  specifies  an  assignment  operation  from  right  to  left. 

Slots  may  have  associated  information  that  specifies  a  default  value,  con¬ 
straints  that  have  to  be  satisfied  by  its  values,  and  exceptions  that  are  to  be  raised 
when  these  constraints  are  not  satisfied,  thus  driving  the  recognition  process.  Excep¬ 
tions  are  necessary  because  they  provide  a  mechanism  for  recording  matching 
failures  during  recognition,  and  are  therefore  used  in  further  processing.  For  instance, 
the  definition  of  the  PERSON  frame  can  be  further  modified  as  follows  : 

frame  PERSON  urith 
prerequisites 

right_leg  :  RIGHT_LEG  such  that 

[  right_leg  connected _ to  _point  torso 

exception  [  NOT_CONNECTED_TO_POINT  urith 
subj  *-  right_leg, 
ref  «-  torso  ]]; 
left_leg  :  LEFT_LEG  such  that 

[  left_leg  connected _ to  _point  torso 

exception  [  NOT_CONNECTED_TO_POINT  urith 
subj  *-  left_leg, 
ref  ■*-  torso  ]]; 

right_arm  :  RIGHT _ ARM  such  that 

[  right_arm  connected _ to_point  torso 

exception  [  NOT_CONNECTED_TO_POINT  urith 
subj  <-  right_arm, 
ref  «-  torso  ]]; 

left_arm  :  LEFT_ARM  such  that 

[  left_arm  connected _ to_point  torso 

exception  [  NOT_CONNECTED_TO_POINT  urith 
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subj  «-  left„arm, 
ref  <-  torso  ]]; 
neck  :  NECK  such  that 

[  neck  connected_to__point  head 
exception  [  NOT„CONNECTED„TO_POINT  with 
subj  «-  neck, 
ref  *-  head  ], 

neck  connected _ to_point  torso 

exception  [  NOT_CONNECTED_TO_POINT  with 
subj  <-  neck, 
ref  *-  torso  ]]; 

head  :  HEAD  ; 
torso  :  TORSO  ; 

end 


The  relationship  ”connected_to_point”  and  several  others  that  will  appear, 
such  as  "behind",  or  other  spatial  relations,  are  borrowed  from  [Badler75].  The  prere¬ 
quisite  slots  above  have  associated  with  them  constraints  that  specify  how  the  parts 
are  inter-connected.  Exceptions  are  raised  whenever  the  specified  constraint  is  not 
verified.  Clearly,  there  are  many  other  necessary  characteristics  for  recognition  of  a 
person  -  we  simply  wish  to  illustrate  the  concepts  here. 

The  exception  NOT_CONNECTED_TO_POINT,  like  all  exceptions,  is  a  frame 

that  is  instantiated  when  a  constraint  is  not  satisfied.  The  NOT_CONNECTED_TO_POINT 

exception  frame  would  look  like  : 

frame  NOT_CONNECTED_TO_POINT  with 
prerequisit  es 

ref  :  FRAME  ; 
subj  :  FRAME  ; 

time_int  :  TIME_INTERVAL  ; 

end 

All  exceptions  are  generated  on  the  occurrence  of  a  matching  failure  of  a  hy¬ 
pothesis  with  the  observed  data,  either  in  a  prerequisite  or  a  dependent  slot.  Some  ex¬ 
ceptions  are  generated  as  system  defaults,  others  are  explicitly  associated  with  slots  in 
frame  definitions.  In  addition,  it  is  not  necessary  for  all  constraints  to  have  explicit  ex¬ 
ceptions  associated  with  them.  The  system  default  exceptions  are  noted  whenever  a 
token  corresponding  to  the  frame  type  of  the  slot  cannot  be  found.  In  other  words,  for 
the  slot  "subj  :  OBJECT",  if  no  OBJECT  token  can  be  found  to  satisfy  all  of  the  con¬ 
straints  specified  for  the  token,  the  exception  raised  would  be  as  if  the  following  were 
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included  in  the  frame  : 


exception  [  NO_SUBJ  with 
subj  «-  self  ] 

where  "self”  refers  to  the  current  frame  token.  In  general,  the  automatic  excep¬ 
tions  are  of  this  form;  however,  the  exception  type  changes  depending  on  the  slot  type 
that  fails  in  matching. 

The  explicit  exceptions  are  of  similar  form  to  the  one  described  above.  They 
describe  various  problems  with  the  observed  data  and  can  have  any  number  of  slots  as¬ 
sociated  with  them. 

The  exception  types  and  the  extra  slots  they  require  will  be  discussed  in  the 
chapter  on  motion  concepts  because  they  are  dependent  on  the  types  of  knowledge  we 
wish  to  represent.  In  addition,  an  exception  propagation  mechanism  is  provided  and 
this  will  also  be  discussed  in  a  later  chapter. 

Another  feature  of  the  formalism  is  the  ability  to  replace  special  variable 
names  with  input  parameters.  The  need  for  this  will  become  clear  in  the  chapter  on 
left  ventricular  motion  concepts.  Such  variable  names  appear  in  upper  case  and  in 
bold  type. 


3.2.2  Primitive  Relationships 


In  addition  to  the  instance_of  relationship,  two  other  primitive  relationships 

are  used  to  structure  a  knowledge  base  into  different  levels  of  abstraction.  The 

PART_OF  relationship  relates  a  frame  to  its  parts,  which  are  the  types  of  its  slots. 

Parts  may  be  decomposable,  such  as 

frame  RIGKT_ARM  with 
prerequisit  es 

lower_r_arm  :  LOWER_RIGHT_ARM  sxlcH  that 

[  lower_r_arm  connected _ to_point  upper_r_arm 

exception  [  NOT_CONNECTED_TO_POINT  with 
subj  <-  lower_r_arm, 
ref  +-  upper_r_arm  ], 
lower_r_arm  connected_to_point  r_hand 
exception  [  NOT_CONNECTED_TO_POINT  with 
subj  <-  lower_r_arm, 
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ref  «-  r__hand  ]]; 

upper„r__arm  :  UPPER__RIGHT _ ARM  ; 

r_hand  :  RIGHT__HAND  ; 

end 

which  breaks  up  the  right  arm  into  three  pieces.  These  slots  in  turn  may  have  con¬ 
straints  and  exceptions  associated  with  them  -  the  formalism  is  recursive  in  this 
respect.  We  can  now  say  that  RIGHT_ARM  is  a  PART_OF  PERSON,  or  conversely,  that 
PERSON  H AS_AS_P ART  a  R1GKT_ARM.  Similarily,  LOWER_RIGHT_ARM  is  PART_OF 
RIGHT_ARM,  etc. 

Generally,  the  PART__OF  relationship  is  a  partial  order  and  defines  a  hierarchy 
of  frames,  the  PART_OF  hierarchy,  where  frames  with  a  primitive  internal  structure 
are  at  the  bottom  and  frames  at  any  other  level  have  the  types  of  their  slots  defined 
by  frames  one  level  down.  Clearly,  recognition  of  an  instance  of  a  frame  makes  heavy 
use  of  the  PART_OF  hierarchy  in  the  recognition  of  its  parts. 

The  other  primitive  relationship  used  to  organize  frames  is  called  the  fS___A 
relationship.  It  is  intended  to  capture  the  notion  of  generalization  in  a  sense  similar  to 
that  in  which  the  concept  of  person  is  a  generalization  of  the  concepts  of  student,  em¬ 
ployee,  and  housewife.  The  use  of  IS_A  economizes  on  frame  size  (through  inheritance 
of  properties),  and  provides  a  mechanism  for  imposing  more  global  constraints  on 
specific  concepts. 

To  illustrate  the  use  of  this  relationship  in  the  organization  of  a  knowledge 
base,  consider  two  specializations  of  PERSON  defined  by: 


frame  ADULT_PERSON  is„a  PERSON  with 
prerequisites 

height  :  such  that 

[  height  ^  4’  exception  [  TOO_SKORT  with 
subj  *-  self  ], 

height  ^  9’  exception  [  TOO_LONG  with 
subj  *-  self  ]]; 


end 


frame  CHILD_PERSON  is_.a  PERSON  with 
prerequisites 

height  :  such  that 

[  height  <  4’  exception  [  TOO_LONG  with 
subj  «-  self  ]]; 

end 

These  two  frames  are  IS _ A  sons  of  PERSON,  and  differentiate  between  adults 
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and  children  (with  the  completely  arbitrary  and  artificial  distinction  of  height).  The 
"height"  slot  is  a  dependent,  and  its  value  is  calculated  from  the  inherited  slots  re¬ 
ceived  from  the  PERSON  frame.  Each  frame  places  restrictions  on  the  value  of  the 
"height"  slot  that  is  acceptable  in  order  for  the  frame  to  be  instantiated.  Therefore, 
adults  must  be  at  least  four  feet  tall,  and  children  under  four  feet  tall.  Note  that  if  a 
person  is  over  nine  feet  tall,  then  perhaps  he  is  not  a  real  person,  and  this  exception 
may  then  be  used  to  activate  the  ANDROID  frame. 

The  IS_A  relationship  is  a  partial  order  relation  and  defines  a  hierarchy  along 
which  frames  can  be  organized.  IS _ A  also  implies  inheritance  of  properties  from  fa¬ 

ther  to  son  nodes  in  the  knowledge  base.  All  slots  in  frame  A  ,  if  A  is  the  father  of  B 

(that  is,  B  IS _ A  A  ),  are  inherited  by  B.  Frame  B  can  refer  to  any  slot  of  A  simply  by 

its  slot  name.  Extra  constraints  may  be  placed  on  the  slot,  or  other  slots  may  be  ad¬ 
ded  in  frame  B  as  long  as  the  IS _ A  relationship  is  still  semantically  valid.  The  hierar¬ 

chy  will  in  general  be  several  levels  deep.  For  example,  ADULT_PERSONS  may  be  divid¬ 
ed  up  into  short  and  tall,  followed  by  thin  and  stout,  followed  by  hair  colour,  eye  colour, 
etc.,  and  all  combinations  of  these. 

We  now  have  enough  representational  tools  with  which  to  tackle  the  problem 
of  combining  inter-image  descriptions  into  descriptions  that  extend  over  several  im¬ 
ages  rather  than  just  two.  Consider  the  use  of  the  IS _ A  and  PART_0F  relationships  in 

modelling  the  following  two  Backus-Naur  production  rules: 

A  =  >  Aa 
A  =>  a 

which  produce  arbitrary  length  strings  of  "a’”s.  This  is  accomplished  with  the  following 

frames  : 

frame  A 
end 

frame  RECURSIVE_A  is_a  A  -with 
prerequisites 

rest  :  A ; 
head  :  UNIT_A  ; 

end 
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frame  UNIT__A  is_  a  A 

end 

According  to  this  definition,  the  A  frame  will  be  a  string  of  instances  of 
UNIT_A ’s.  Let  us  relate  this  now  to  the  inter-image  description  problem.  In  instantiat¬ 
ing  a  motion  concept,  a  general  definition  of  say  "lengthen”,  would  be  that  the  size  of 
an  object’s  longitudinal  axis  increases.  Therefore,  as  soon  as  it  increases,  and  the  in¬ 
crease  is  recognized,  the  frame  would  have  all  of  its  prerequisites  satisfied  and  should 
be  instantiated.  That  instance,  however,  does  not  necessarily  give  the  entire  time  in¬ 
terval  over  which  the  lengthening  occurred.  The  "lengthen"  frame  would  then  be  in¬ 
stantiated  for  every  pair  of  images  for  which  the  length  of  the  object  increased  (the 
string  of  ”a"’s).  A  mechanism  is  needed  for  combining  these  into  a  single  "lengthen" 
frame  (A).  The  properties  of  IS _ A  permit  us  to  do  this.  Frame  A  would  define  the  con¬ 

cept  for  an  arbitrary  time  duration.  Frame  UNIT_A  would  inherit  the  properties  of  A, 
that  is  the  definition  of  the  concept,  but  would  place  a  restriction  on  the  duration  -  the 
event  must  have  duration  of  only  one  time  unit  (the  time  interval  between  successive 
images).  Frame  RECURSIVELY  has  as  parts  one  slot  of  type  UNIT_A  and  one  of  A,  thus 
increasing  the  time  interval  of  the  event  by  one  image  pair  on  every  instance.  The  final 
instance,  in  time,  would  be  the  one  that  describes  the  entire  interval  over  which  the 
event  occurred.  This  mechanism  is  an  implicit  one,  built  into  the  instantiation  pro¬ 
grams;  however,  it  can  be  represented  formally  in  the  knowledge  base  as  shown  above. 
All  frames  refer  to  the  motion  frame  corresponding  to  the  A  frame  above,  while  all  low 
level  inter-image  descriptions  refer  to  the  UNIT_A  frame. 

3.2.3  Similarity  Links 

Our  representation  scheme  is  clearly  biased  towards  recognition.  For  in¬ 
stance,  the  classification  of  slots  into  prerequisites  and  dependents  is  useful  to  a  recog¬ 
nizer  which  has  to  know  which  portions  of  a  frame  definition  must  be  observed  in  the 
data  and  which  portions  define  how  to  instantiate  the  frame  concept. 


-  36  - 


Perhaps  the  most  important  representational  feature  for  the  purposes  of 
recognition  is  the  similarity  link.  These  links  relate  frames  that  are,  in  some  sense, 
similar,  and  are  used  to  handle  situations  where  one  or  more  matching  exceptions  have 
been  raised.  Frames  that  have  an  active  similarity  link  between  them  are  assumed  to 
be  mutually  exclusive.  The  information  associated  with  a  similarity  link  is  stored  inside 
a  generic  frame.  The  link  remains  dormant,  i.e.,  the  source  frame  and  the  specified 
destination  frame  are  not  connected  in  any  way,  until  the  recognition  process  deems 
the  connection  necessary. 

The  features  of  similarity  links  as  used  in  our  representation  scheme  have 
their  roots  in  the  exception  handling  mechanisms  proposed  in  [Goodenough?5]  and 
[Wasserman77],  but  are  closer  to  those  used  in  [Mylopoulos78],  Note  that  similarity 
links,  as  defined  here,  depend  strongly  on  the  PART__OF  hierarchy,  since  the  similarity 
links  of  a  frame  A  are  used  to  handle  exceptions  which  are  raised  during  the  recogni¬ 
tion  of  the  parts  of  A. 

Each  similarity  link  carries  two  types  of  information  (in  addition  to  its  source 
frame  and  destination  frame(s)): 

•  A  condition  which  determines  whether  the  raised  exceptions  can  indeed  be 
handled  by  the  link;  and 

•  binding  information  that  indicates  how  the  values  of  the  slots  in  the  source 
frame  are  related  to  those  in  the  destination  frame. 

The  condition  is  made  up  of  two  components,  a  similarity  expression  ,  and 
a  difference  expression.  The  similarity  expression  contains  the  necessary  conditions 
which  the  two  linked  frames  must  share  in  order  for  the  similarity  link  to  be  activated 
(i.e.,  the  "similarities").  Since  only  one  of  the  frames  (the  source  frame)  is  active  when 
this  condition  is  evaluated,  the  predicates  describe  prerequisite  components  of  the 
destination  frame  that  must  also  be  possessed  by  the  source  frame.  The  difference  ex¬ 
pression  is  a  list  of  exceptions  which  may  be  raised  during  matching  of  the  source 
frame.  Each  of  the  list’s  elements  has  an  associated  time  interval  that  specifies  when 
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each  exception  should  occur  if  all  of  them  are  observed  in  the  image  sequence.  This 
time  interval  may  either  be  present  as  a  slot  of  type  TIME_INTERVAL  in  the  exception 
token,  or  as  a  time  instant  value  using  the  construct  described  in  Appendix  A.  In 
effect,  such  an  expression  indicates  the  time  course  of  the  matching  failures  in  the 
source  frame  context,  that  indicate  that  the  destination  frame's  context  would  be 
more  relevant  (i.e.,  the  "time  sequence  of  differences").  Thus,  the  similarity  links  point 
to  alternate  frames  potentially  viable  when  exceptions  are  raised. 

An  activated  frame,  when  it  activates  a  similar  frame,  may  have  certain  al¬ 
ready  instantiated  data  which  may  be  of  use  to  the  destination  frame.  The  simlarity 
link  therefore  may  specify  bindings  between  the  linked  frames.  The  binding  is  accom¬ 
plished  at  the  slot  name  level  with  the  notation  being  : 

source_slot_name  ->  destination_slot-name 

It  is  not  always  true  that  such  bindings  are  possible  and  in  many  cases  re¬ 
calculations  are  necessary. 

To  illustrate  the  use  of  similarity  links,  consider  a  modification  of  the  PERSON 


frame. 

frame  PERSON  with 
prerequisites 
/*  as  above  */ 
dependents 
/*  as  above  */ 

similarity  links 

sim_linkl  :  ANDROID_UNDER_ REPAIR 

for  differences  : 

dl  :  NOT_CONNECTED_TO_POINT  where 

[  subj  instance_of  L0WER_R1GKT _ ARM, 

ref  instance_of  R_HAND  ]; 

bound  by  : 

right_leg  -»  right_leg  , 
loft_leg  ->  left_leg  , 
torso  ->  torso  , 
left_arm  -»  left_arm  , 
neck  ->  neck  , 
head  -*  head  ;; 

end 


For  another  example,  suppose,  that  for  some  action,  the  subject  of  the  action 
must  be  an  adult,  (such  as  going  to  a  height  restricted  ride  at  a  fair  ground).  There- 
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fore,  a  part  of  this  event,  the  subject  slot,  must  be  of  type  ADULT_PERSON.  If  the  PER¬ 
SON  frame  raises  the  exception  TOO_SHORT,  then  the  parent  frame,  ADMIT_TO_RIDE 
should  have  a  similarity  link  to  handle  it.  This  frame  would  be  : 

frame  ADMIT_TO_RIDE  with 
prerequisit  es 

agent  :  ADULT_PERSON  ; 
obj  :  AEULT_PERSON  ; 
similarity  links 

sim_linkl  :  DO_NOT_AEMIT 
for  differences  : 

dl  :  T00_3K0RT  where 

[  subj  instance_of  obj  ]; 
bound  by  : 

obj  -»  obj  , 
agent  ->  agent  ;; 

end 

It  is  clear  that  the  parent  frame  (along  the  PART_OF  hierarchy)  always  han¬ 
dles  exceptions.  That  frame  alone  knows  the  context  within  which  the  recognition  is 
being  done  and  only  it  can  therefore  properly  decide  what  action  should  be  taken. 

Similarity  links,  in  addition  to  frame  slots,  are  inherited  along  the  IS_A 
hierarchy.  This  feature  is  very  useful  because  it  drastically  reduces  the  number  of 
links  that  have  to  be  specified  explicitly  in  the  knowledge  base,  since  it  will  not  always 
be  true  that  explicit  similarity  links  are  provided  for  all  the  kinds  of  exceptions  which 
may  occur. 

An  additional  mechanism  is  necessary  for  the  preservation  of  context  when  a 
similar  frame  is  activated.  This  mechanism  is  dependent  on  the  PART_OF  hierarchy 
and  on  exception  propagation.  Exceptions,  therefore,  in  addition  to  the  slots  men¬ 
tioned  above,  must  also  have  a  special  slot  to  aid  in  this  task.  Such  slots  are  named 
"prereq_part”.  This  mechanism  will  be  described  in  detail  in  a  later  chapter.  For  now, 
consider  the  following  example.  Walking  can  be  described  as  combination  of  transla¬ 
tion  and  arm  and  leg  swinging  motions  (the  frames  for  walking  will  be  given  in  the  next 
chapter).  In  turn,  the  swinging  motions  have  as  components  the  various  motions  exhi¬ 
bited  by  the  arms  and  legs.  These  motions  also  have  frames  associated  with  them 
which  describe  the  motions  of  the  parts  of  each  leg  and  arm.  Now,  if  the  lower  right 
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forearm  and  hand  do  not  exhibit  the  proper  motion,  because  they  are  in  a  cast,  an  ex¬ 
ception  will  be  raised.  This  exception  will  be  handled  by  its  parent  frame  which  would 
be  the  frame  for  right  arm  swings,  say  forward  swings.  It  may  activate  the  "impaired 
right  arm  forward  swing"  frame,  for  example.  This  exception  would  then  propagate  up 
the  PART_OF  hierarchy  so  that  the  proper  context  for  the  "impaired  right  arm"  could 
be  found.  The  PART_OF  hierarchy  for  this  example  would  look  like  that  in  Figure  3.1. 
The  process  of  propagation  and  of  placing  constraints  on  which  parent  frame  should  be 
activated  would  lead  to  the  activation  of  the  "impaired  right  arm  walk"  frame. 

3.3  A  Simple  Object  Knowledge  Base 

Consider  a  knowledge  base  (KB)  for  points,  line  segments,  and  closed  curves  - 

a  very  simple  KB,  but  one  that  is  needed  for  the  discussion  in  the  next  two  chapters. 

The  frame  for  objects,  that  is,  the  root  frame  in  the  object  IS_A  hierarchy,  contains  the 

properties  that  all  objects  have  a  motion  capability  and  a  shape  classification  : 

frame  OBJECT  with 
prerequisites 

motion_cap  :  CAPABILITY  ; 
shape_type  :  SHAPE  ; 

end 

Shape  types  and  motion  capability  types  have  procedural  definitions  that  are 
provided  by  the  user  and  incorporated  into  the  image  analysis  component  of  the  sys¬ 
tem.  The  frame  for  a  point  is  straightforward  : 

frame  POINT  is_ a  OBJECT  with 
prerequisites 

x : NUMBER ; 
y : NUMBER ; 
shape_type  :  DOT  ; 

end 

where  "x"  and  "y"  are  the  co-ordinates  in  the  plane.  Note  that  the  frame  does  not 
specify  whether  or  not  the  point  is  "fixed"  (cannot  move),  "mobile"  (can  move  if  propul¬ 
sion  is  provided  from  an  outside  source),  "motile"  (can  move  under  its  own  power), 
"contractile"  (motile  plus  capabitility  to  change  shape),  etc.  A  line  segment  has  two 
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points  as  parts,  a  start  point  and  an  end  point.  Because  of  these,  the  additional  pro¬ 
perty  of  arclength  can  be  added.  LINE_SEG  looks  like  : 


frame  LINE_SEG  is _ a  OBJECT  with 

prerequisites 

start_pt  :  POINT  ; 
encL-pt  :  POINT  ; 
shape_type  :  LINE  ; 
dependents 

arclen  :  LENGTH_V  ; 


end 


Note  that  a  line  segment  is  made  up  of  a  series  of  adjacent  points.  All  of 

these  points  may  be  included  in  the  above  frame,  or  only  a  representative  set  of  them 

may  be  included.  In  either  case,  several  additional  slots  would  be  included  in  the  above 

frame,  one  for  each  point,  and  all  would  be  of  type  POINT.  An  open  line  segment,  that 

is,  one  that  does  not  enclose  a  finite  region,  has  a  frame  defined  by: 

frame  OPEN_LINE_SEG  is_a  LINE_SEG  with 
prerequisites 

start_pt  :  such  that  [  start_pt  ~same_as  end _ pt  ]  ; 

end 


"same_as”  is  a  relationship  that  checks  for  approximate  equality  between  its 

arguments,  depending  on  the  type  of  quantities  being  compared.  For  the  straight  line 

frame,  we  can  add  in  an  explicit  calculation  for  the  arclength  and  orientation  : 

frame  STR_LINE  is_ a  0PEN__L1NE_SEG  with 
prerequisit  es 

shape_type  :  STRAIGHT  ; 
dependents 

arclen  :  with 

arclen  *-{{start__pt.x  —  end^jpt.  x  )2  +  (start _pt.y  —endp^jpt.y  )2)0'5; 
orient  :  ANGLE_V  with 

orient  «-  arctan((end_pt.y  -  start_pt.y)  4-  (end_pt.x  -  start_pt.x))  ; 

end 

Special  types  of  straight  lines  are  the  lateral  and  longitudinal  axes  of  closed  line  seg¬ 
ments. 

frame  LATERAL_AJX1S  is_a  STR_LINE  with 
prerequisit  e  s 

start_pt  :  such  that 

[  for  line 3  :  LONGITUDINAL_AXIS  where 

[  angle(line3.start_pt,  obj  1. centroid,  self.start_pt)  =  rr/2  , 
self.start_pt  part_of  objl  , 

self. end _ pt  part_of  objl  , 

~(  for  line2  :  STR_LINE  where 
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[  line2. arclen  >  self.arclen  , 

line2.end _ pt  part_of  objl  , 

line2.start_pt  part_of  objl  , 

angle(line3.start_pt,  obj  1. centroid,  line2.start_pt)  =  rr/ 2  ]  )  ]]  ; 

dependents 

intersect_pt  :  POINT  with 

intersect_pt  «-  ptl  such  that 
[  for  ptl  :  POINT  where 
[  pi  part_of  Iine3  , 
pi  part_of  self  ]  ]  ; 

end 

frame  LONGITUDINAL-AXIS  is_a  STR_LINE  with 

prerequisites 

arclen  :  such  that 

[  for  objl  :  CL03ED_LINE_SEG  where 
[  obj  1. centroid  part_of  self  , 
self.start_pt  part_of  objl  , 

self. end _ pt  part_of  objl  , 

for  linel  :  STR_LINE  where 
[  linel. arclen  >  self.arclen  , 
line  1. start_pt  part_of  objl  , 

linel. end _ pt  part^of  objl  , 

objl. centroid  part__of  linel  ]  )  ]]; 

end 

frame  CURVED_SEG  is_a  OPEN_LINE_SEG  with 
prerequisites 

depth _ pt  :  POINT  ; 

mouth_len  :  LENTGH_V  ; 

depth  :  LENTGK _ V  ; 

shape_type  :  CURVE  ; 

end 

Closed  outlines,  such  as  left  ventricles,  have  several  properties  :  a  centroid, 
an  arclength,  an  area,  and  longitudinal  and  lateral  axes.  Also,  left  ventricles  have  mo¬ 
tion  capabilities  of  "contractile”,  while  objects  such  as  ribs,  or  the  diaphragm  are 
"fixed"  with  respect  to  the  image.  The  closed  outline  frame  is  : 

frame  CLOSED_LINE_SEG  is_ a  LINE— SEG  with 
prerequisites 

start_pt  :  such  that  [  start_pt  same_as  end _ pt  ]  ; 

centroid  :  POINT  ; 
long_axis  :  LONGITUDINAL-AXIS  ; 
lat_axis  :  LATERAL-AXIS  ; 
area  :  AREA_V  ; 

end 

while  the  frames  for  contractile  and  fixed  objects  are  : 

frame  CONTRACTILE_OBJECT  is_a  CLOSED_LINE_SEG  with 
prerequisites 

motion _ cap  :  CONTRACTILE  ; 

end 
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frame  FXXED_OBJECT  is._a  CLOSED_LINE_SEG  with 
prerequisit  es 

motion_cap  :  FIXED  ; 

end 


-43- 


CHAPTER  4 


MOTION  CONCEPTS 

4.1  Semantic  Components  for  Motion  Concepts 

In  natural  language,  the  tools  used  for  description  of  motion  concepts  are 
verbs  such  as  cross,  lift  and  run;  directional  prepositions  such  as  towards,  through  and 
over;  and  adverbs  such  as  slowly  or  quickly.  In  order  for  these  concepts  to  make 
sense  in  a  sentence,  certain  semantic  components  must  be  present.  These  com¬ 
ponents  completely  define  the  motion  concept.  Miller  [Miller72]  has  done  a  very  good 
analysis  of  the  semantic  components  for  a  large  class  of  English  motion  verbs.  Using 
Miller’s  work  as  a  foundation,  Badler  [Badler75]  refined  these  concepts,  gave  detailed 
definitions  of  the  directionals,  and  outlined  a  framework  within  which  recognition  can 
be  done.  Tsotsos  [Tsotsos76]  further  investigated  the  semantics  and  representation  of 
these  motion  concepts.  The  work  presented  here  is  strongly  based  on  these  three 
research  efforts. 

The  complete  list  of  Miller’s  components  along  with  their  meanings  is  sum¬ 
marized  below; 

motion  —  a  subject  Y  changes  location 

reflexive-objective  --  reflexive  pronoun-direct  object 

causative  —  an  agent  Y  causes  an  object  X  to  change  location 

permissive  —  an  agent  Y  allows  an  object  X  to  change  location 

propellant  —  an  agent  Y  applies  force  to  an  object  X  so  that  X  changes  location 

directional  --  a  subject  Y  changes  location  in  a  specified  direction 
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medium  —  a  subject  Y  changes  location  in  some  medium 

instrumental  —  an  agent  Y  using  an  instrument  Z  causes  X  to  change  location 
inchoative  --  a  subject  Y  changes  one  of  its  physical  properties 
change-of-motion  —  a  subject  Y  changes  its  current  state  of  motion 
deictic  —  a  subject  Y  changes  location  toward  or  away  from  the  speaker 
velocity  —  a  subject  Y  changes  location  with  some  velocity. 

Miller  did  his  analysis  from  a  linguistic  point  of  view,  using  natural  language. 
Components  such  as  "propellant"  or  "permissive”,  which  describe  the  intent  of  the 
agent  of  the  action,  may  be  impossible  to  detect  from  a  sequence  of  images,  "speak¬ 
er"  is  non-existent  but  may  be  interpreted  as  "observer".  We  assume  that  the  observer 
plays  no  role  in  the  motion  description  process.  In  addition,  he  did  not  provide  details 
on  the  components  for  the  description  of  the  changes  in  an  object’s  observable  physi¬ 
cal  properties  such  as  "expansion”  or  "joining  two  objects  to  form  one".  Miller’s  com¬ 
ponents  only  describe  motions  of  rigid  or  jointed  objects. 

Let  us  now  re-define  these  components  and  place  them  in  a  form  which  is 
unambiguous  for  a  visual  motion  understanding  system. 

Motion  -  An  object,  of  any  type,  has  exhibited  a  motion  if  a  change  of  location 
has  been  observed  for  its  centroid,  or  its  parts.  This  definition  may  be  applied  recur¬ 
sively  to  any  of  the  object’s  parts. 

Reflexive-Objective  -  The  distinction  between  reflexive  and  objective  verbs 
or  directionals  depends  on  the  reference  point  for  the  descriptive  term  used.  If  the 
reference  is  the  computer  imposed  co-ordinate  axes,  then  the  descriptive  term  to  be 
used  is  reflexive.  Examples  of  reflexive  terms  are  the  directionals  "upwards"  and  "left¬ 
wards"  or  the  verbs  "rise"  and  "fall".  If  the  description  is  in  terms  of  some  other  ob¬ 
ject,  whether  it  is  in  motion  or  not,  then  the  term  to  be  used  is  objective.  Examples  of 
objective  directional  prepositions  are  "towards"  or  "over",  while  verbs  are  "cross"  or 
"approach”. 

Causative  -  Miller’s  definition  is  difficult  to  deal  with  in  a  vision  system. 
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Causation  frequently  requires  knowing  the  intent  of  the  object  causing  the  motion.  In 
addition,  the  properties  of  the  problem  domain  must  be  very  well  understood  in  order 
to  be  able  to  model  them  at  a  deep  enough  level  to  include  causative  relationships. 
Causative  components  will  not  be  included  in  our  system. 

Permissive  -  Again,  the  definition  Miller  provides  implies  intent.  However, 
several  verbs  such  as  "drops”  or  "releases"  do  not  require  intent  in  order  to  be  recog¬ 
nized.  Such  motions  require  complex  semantics:  the  agent  of  the  action  must  initially 
be  restraining  the  object  (that  is,  referent)  of  the  motion  so  that  the  object  cannot 
move  in  some  specific  direction  or  with  some  specific  velocity.  The  agent  then  exhibits 
some  motion,  which  is  dependent  on  the  agent’s  type,  that  frees  the  object  to  exhibit 
some  motion  that  it  previously  could  not  exhibit. 

Propellant  -  The  application  of  force  can  be  precisely  defined  for  a  vision 
system  so  that  agent  intent  or  forces  such  as  gravity  or  magnetism  need  not  be  con¬ 
sidered.  The  participating  objects  must  be  in  contact,  at  least  one  of  them  must  be  ca¬ 
pable  of  self -propulsion  (have  a  motion  capability  of  motile),  and  the  resulting  motion 
of  the  non-motile  object  is  in  the  same  direction  as  the  motion  of  the  motile  object  that 
applies  the  force.  Examples  of  such  verbs  are  "lift",  "lower",  "push",  "pull",  "shove"  or 
"drag". 


Directional  -  Directionals,  as  mentioned  previously,  have  been  adequately 
defined  by  Badler  [Badler75j;  his  definitions  will  be  used  with  little  or  no  modification. 
For  example,  the  definition  of  the  objective  directional  "across"  involves  the  following 
sequence  of  changes  in  spatial  relations  between  the  two  participating  objects: 
object  A  is  moving,  object  B  is  fixed, 
object  A  nears  object  B. 

object  A  is  above  object  B  on  a  particular  side  C  of  it. 

object  A  is  above  object  B  on  another  side  of  B  which  is  opposite  to  C. 

object  A  is  no  longer  above  B,  but  is  still  near  it. 

object  A  is  no  longer  near  object  B. 
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In  general,  all  objective  directional  definitions  involve  a  sequence  of  changes  in  the  spa¬ 
tial  relations  between  the  participants. 

Medium  -  The  medium  here  defines  the  element  in  which  the  motion  is  taking 
place  whether  it  be  on  land,  in  the  air,  in  the  water  or  in  a  patient’s  chest.  For  our  ap¬ 
plication  domain,  it  is  clear  that  the  medium  is  in  a  patient’s  chest.  Therefore,  the 
medium  will  not  be  explicitly  referred  to,  but  the  methodolgy  could  take  care  of  it  if 
the  problem  domain  required  it. 

Instrumental  -  Instruments  are  used  in  causing  motions.  Actions  such  as 
using  a  crowbar  as  a  lever  or  a  chain  as  a  connection  between  two  cars  are  examples  of 
such  motions.  It  has  already  been  stated  that  causative  components  will  not  be  includ¬ 
ed  in  our  system  and  therefore  neither  will  the  instrumental  component. 

Inchoative  -  This  defines  changes  of  an  object’s  condition  or  spatial  relations. 
An  example  is  "The  door  opened"  -  rephrased  so  that  the  inchoative  component  be¬ 
comes  more  clear,  "The  door  became  open".  The  change  described  by  the  verb  "open" 
is  that  the  size  of  the  aperture  between  the  door  proper  and  the  door  jamb  becomes 
larger.  The  form  of  the  inchoative  component  is  generally  "becomes  <adjective>",  and 
in  many  cases,  single  verbs  may  be  found  that  describe  the  change.  Such  changes  in 
physical  properties  or  spatial  relations  are  not  well  defined  by  Miller  and  will  be  ex¬ 
panded  upon  below. 

Change- of -motion  -  This  component  describes  not  only  the  beginning  or  the 
ending  of  a  motion,  but  also  the  change  from  one  motion  class  to  another  for  the  same 
object  when  the  two  motions  are  adjacent  in  time.  This  component  appears  only  impli¬ 
citly  as  the  start  and  end  time  relationships  for  any  motion  and  the  constraints  on 
which  motions  may  preceed  or  follow  it.  Instances  of  this  component  are  determined 
by  the  temporal  segmentation  process. 

Deictic  -  Observers,  as  stated,  earlier,  are  not  participants  in  our  system. 
The  main  reason  why  they  are  not  included  is  that  motions  such  as  "towards"  or  "away 
from"  the  observer  require  recognition  of  changes  in  the  distance  from  the  object  to 
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the  observer  -  in  other  words,  depth  perception.  Our  system  considers  only  2D  mo¬ 
tions  at  present. 

Velocity  -  Velocity  terms  such  as  "slowly”  or  "quickly"  are  used  in  relation  to 
the  object’s  motion  capabilities.  For  example,  a  car  moving  at  30  km/hr  is  moving 
slowly,  while  that  speed  for  a  person  is  quite  fast.  Thus,  if  velocity  descriptions  are  re¬ 
quired,  they  must  be  defined  individually  for  every  class  of  objects  in  the  system’s 
domain. 

Our  knowledge  representation  was  designed  to  accomodate  all  of  the  above 
motion  semantic  components,  except  for  those  involving  causation  or  three- 
dimensional  movements.  These  components  appear  in  our  knowledge  base  in  diverse 
ways.  If  an  object  exhibits  a  motion  ,  then  some  frame  within  the  motion  hierarchy  is 
instantiated  to  represent  the  motion,  with  the  subject  of  the  motion  being  the  object 
which  exhibits  the  motion.  Subjects  will  always  appear  under  the  slot  name  "subj".  In 
addition  to  the  "subj"  slot,  there  is  always  a  time  interval  during  which  the  motion  oc¬ 
curs.  This  slot  is  always  named  "time_int”  and  is  always  of  type  TIME_INTERVAL,  which 
will  be  defined  later.  Reflexive  and  objective  motions  are  distinguished  by  the  ab¬ 
sence  or  presence  respectively  of  a  referent  for  the  motion.  The  "ref'  slot  denotes 
the  object  to  which  the  motion  is  compared.  Permissive  or  propellant  components 
do  not  appear  explicitly,  but  simply  define  different  classes  of  example  verbs.  Such 
verbs  have  both  an  agent  which  appears  under  the  "agent"  slot  and  represents  the  ob¬ 
ject  which  is  producing  the  motions,  and  an  object  which  appears  under  the  "obj"  slot 
and  represents  the  object  which  is  being  affected  by  the  motion  of  the  agent.  Direc- 
tionals  are  used  in  two  ways  -  numerical  trajectories  or  qualitative  directional  terms. 
If  a  numerical  trajectory  is  used  to  define  a  motion,  it  appears  under  the  slot  name 
"traj",  while  directionals  always  use  the  slot  name  "dir".  Velocity  is  included  as  the 
slot  "speed".  Inchoative  components  will  be  discussed  below  along  with  how  they  are 
represented  in  our  knowledge  base.  The  change- of -motion  component,  as  explained 
above,  appears  as  timing  constraints.  If  one  motion  A  must  immediately  preceed 
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another  B  ,  then  the  constraint  is: 


A.time_int.encL_time  =  B.time_J.nt.start_time 

All  motions  have,  "start_tLme”  and  "end _ time”  slots  (since  these  are  part  of  the 

TIME_INTERVAL  frame),  and  thus  for  any  motion  the  frame  may  define  the  relative 
start  and  end  time  of  the  motion  and  any  constraints  on  which  motions  may  preceed  or 
follow  it. 

The  inchoative  component  will  now  be  discussed  in  detail.  This  component 
describes  changes  in  the  physical  properties  of  an  object  or  the  spatial  relations  it  ex¬ 
hibits  with  the  rest  of  the  world.  Concepts  for  the  change  of  an  object’s  physical  pro¬ 
perties  are  particularly  necessary  for  our  application  domain  of  left  ventricular  wall 
motion.  Let  us  begLn  by  first  considering  what  types  of  properties  are  relevant  for  con¬ 
tractile  objects  (such  as  LV’s): 

Shape  -  both  global  shape  and  the  shape  of  any  definable  sub-parts  of  the  object. 

Size  -  area,  length  (size  of  longitudinal  axis),  width  (size  of  lateral  axis),  and  per¬ 
imeter  (arclength). 

Location  -  displacements  of  points  or  features  of  the  object. 

The  next  task  is  to  define  appropriate  motion  concepts  for  changes  in  these 
properties.  Location  changes  have  already  been  defined.  Shape  is  a  very  difficult  con¬ 
cept  to  deal  with  satisfactorily.  Other  researchers  have  used  various  techniques  to  en¬ 
code  the  boundary  of  a  closed,  simple  outline  as  a  list  of  protrusions,  line  segments, 
corners,  etc.  [Pavlidis78j.  For  left  ventricles,  "shape  index"  defined  by 
47rarea/perimeter  has  been  used  in  the  past  [Gibson75].  This  particular  description  is 
hard  to  deal  with  and  does  not  provide  a  unique  descriptor  for  a  unique  shape.  The 
shape  description  method  that  we  use  was  intuitively  derived  by  studying  cardiologists’ 
definitions  of  LV  motions.  It  is  quite  similar  to  that  found  in  [Pavlidis79a]. 

Consider  first  the  kinds  of  motion  exhibited.  The  segments  of  the  LV  wall  can 
change  length,  change  shape,  and  move  arbitrarily  in  the  plane  of  the  image.  In  addi¬ 
tion,  it  is  not  clear  which  points  to  follow  from  image  to  image.  Experience  in  observ- 
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ing  cardiologist’s  analyses  and  descriptions  led  to  the  belief  that  the  human  eye  cannot 
follow  single  points  on  such  a  time-varying  dynamic  outline.  However,  the  eye  does 
seem  to  be  able  to  follow  points  when  additional  structure  due  to  local  shape  is  added. 
That  is,  a  bump  on  a  smooth  outline,  regardless  of  changes  in  location,  orientation,  ar- 
clength  or  size  can  be  followed  from  image  to  image  and  can  still  be  identified  as  the 
same  bump.  Cases  in  point  are  the  papillary  muscles,  the  edges  of  the  aortic  and  mi¬ 
tral  valve,  and  the  apex.  Such  observations,  together  with  consideration  of  the  ways 
most  naturally  used  by  both  humans  and  the  algorithms  that  we  wished  to  develop, 
have  led  to  the  following  shape  characterization.  The  LV  outline  is  described  in  terms 
of  concave  and  convex  segments.  In  addition,  these  shape  features  can  be  further 
sub-classified  in  terms  of  geographic  coastline  features  where  the  interior  of  the  LV  is  a 
lake,  and  walls  are  the  shores  of  the  lake.  Shape  features  have  associated  descriptors 
such  as  sixes,  axis  lengths,  arclength,  etc.  A  number  of  shape  types  have  been  defined 
and  will  be  discussed  in  detail  below.  For  complex  shaped  outlines,  it  may  be  neces¬ 
sary  to  derive  a  hiersirchy  of  shape  description,  which  gives  large,  gross  features  at  one 
level  and  then  the  smaller,  more  detailed,  features  within  them  at  a  lower  level.  More 
details  are  given  in  Chapter  6. 

Globed  shape  is  more  difficult  to  deal  with.  One  can  define  a  set  of  common 
smooth  closed  outline  shapes  and  then  construct  a  recognizer  for  such  shapes  which 
can  find  the  best  matches  [Agarwal78].  Such  shape  types  include  :  kidney,  lens,  cres¬ 
cent,  heart,  drop,  horseshoe,  circle,  oval,  ellipse.  These  shape  types  also  have  longitu¬ 
dinal  and  lateral  axes  as  well  as  arclength  and  a  centroid.  Left  ventricles  do  not  exhibit 
such  a  wide  range  of  possible  shapes  as  listed  above.  Indeed,  no  formal  characteriza¬ 
tion  of  global  LV  shape  has  seemed  useful  to  cardiologists  other  than  "roundedness". 
Left  ventricles  have  an  "LV  shape"  (ellipsoid)  if  they  are  healthy  and  are  functioning 
normally,  and  may  become  more  "rounded"  (circular)  or  more  "elongated"  (cylindri¬ 
cal)  in  abnormal  situations.  Therefore,  as  far  as  global  shape  descriptors  go,  only  such 
very  gross  shape  types  will  be  considered  for  left  ventricles. 
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Size  and  shape  properties  may  change  with  time.  Axes  change  length  when 
the  object  they  belong  to  contracts  or  expands.  Since  motion  is  only  two  dimensional, 
we  do  not  worry  about  foreshortening  effects  due  to  rotation  into  the  image  plane. 
Axes  also  change  orientation,  but  such  changes  are  handled  by  another  class  of  mo¬ 
tions.  Also,  when  an  object  contracts  or  expands,  its  area  and  arclength  (perimeter) 
changes.  Shape  types  may  change  from  one  to  another.  The  description  of  these 
changes  is  the  most  difficult  of  all.  When  a  smooth  relatively  featureless  contour  is  al¬ 
lowed  to  change  virtually  without  restriction,  the  shape  combinations  become  un¬ 
manageable.  Therefore,  only  a  few  types  of  shape  changes  will  be  handled  by  explicit 
terms.  A  "catch-all"  term  will  be  provided  for  the  remainder.  For  our  domain  this  is 
reasonable  because  of  continuity  of  motion  assumption  and  because  the  physical  pro¬ 
perties  of  the  left  ventricle.  This  will  be  further  discussed  in  a  subsequent  section. 

We  now  have  a  complete  set  of  the  basic  descriptors  for  motion  of  contractile 
objects:  changes  in  location,  size,  and  shape.  There  are  four  primitive  frames  in  our 
knowledge  base  for  these  concepts  and  they  are  used  in  abstraction  to  all  higher  level 
terms.  First  however,  the  frame  for  a  time  interval  is  given  since  all  changes  are 
described  in  terms  of  the  time  interval  in  which  they  occur. 

frame  TIME_INTERVAL  with 
prerequisites 

start_time  :  TIME_V  such  that  [  start_time  ^  0  ]; 
end__time  :  TIME_V  such  that  [  end _ time  >  start_time  ]; 

dependents 

duration  :  TIME_V  with  duration  <-  end _ time  -  start_time  ; 

end 

The  four  primitive  frames  AREA _ CHANGE,  L0CAT10N_CHANGE,  LENGTH_CHANGE,  and 

SHAPE_CHANGE  follow: 

frame  AREA__CHANGE  with 
prerequisites 

subj  :  CONTRACTILE_OBJECT  ; 
time_int  :  TIME_INTERVAL  ; 

start_a  :  AREA _ V  ; 

end _ a  :  AREA _ V  ; 

end 

frame  L0CAT10N_CHANGE  with 
prerequisites 
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subj  :  POINT  such  that 

[  for  obi  :  OBJECT  "where 

[  obl.mot_cap  instance_of  MOBILE  , 
subj  part_of  obi  ]]; 
time_int  :  TIME_INTERVAL  ;  " 
start_loc  :  POINT  ; 
end _ loc  :  POINT  ; 

end 

frame  LENGTH _ CHANGE  -with 

prerequisites 

subj  :  LINE_SEG  ; 
time_int  :  TIME_INTERVAL  ; 
start_l  :  LENGTH_V  ; 
end _ 1  :  LENGTH_V  ; 

end 

frame  SHAPE_CHANGE  with 
prerequisites 

subj  :  C ONTRACTI LE_0 B JE CT  ; 
start_shape  :  SHAPE  ; 

end _ shape  :  SHAPE  ; 

time_int  :  TIME_INTERVAL  ; 

end 

These  all  use  other  primitive  frames  such  as  OBJECT,  POINT,  various  data  types 

(LENGTH_V,  TIME_V,  AREA_V,  CHAR_STRING). 

Higher  level  concepts  can  now  be  defined  using  these  five  basic  frames.  For 

example,  the  AREA_CHANGE  frame  leads  us  to  the  concept  of  contraction: 

frame  CONTRACT  is_a  AREA_CHAXGE  with 
prerequisites 

start_a  :  such  that 

[  start_a  >  end _ a  ]; 

dependents 

speed  :  SPEED_V  with 

speed  «-  (start _ a  -  end _ a)  -r  time-int. duration  ; 

end 

frame  EXPAND  is_a  AREA_CHANGE  with 
prerequisites 

start_a  :  such  that 

[  start_a  <  end _ a  ]; 

dependents 

speed  :  SPEED_V  with 

speed  (end _ a  -  start_a)  -r  time_int. duration  ; 

end 

LENGTK__CHANGE  can  be  further  specialized  by  verbs  such  as  compress,  narrow,  widen 
and  lengthen: 

frame  WIDEN  is_a  LENG'TH_ CHANGE  with 
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prerequisites 

subj  :  LATERAL__AXIS  ; 
start_l  :  such  that 

[  start_l  <  end _ 1  ]; 

dependents 

subj  :  OBJECT  urith 

subj  «-  super_part(subj)  ; 
speed  :  SPEED_V  with 

speed  *-  (end_l  -  start_l)  ~  time_int. duration  ; 

end 


”super_partM  is  a  function  that  returns  the  concept  of  which  its  argument  is  a 

part  of  (the  inverse  of  the  PAJRT_OF  relation). 

frame  NARROW  is_a  LENGTH- CHANGE  with 
prerequisites 

subj  :  LATERAL_AXIS  ; 
start_l  :  such  that 

[  start_l  >  end _ 1  ]; 

dependents 

subj  :  OBJECT  with 

subj  <-  super_part(subj)  ; 
speed  :  SPEED_V  with 

speed  *-  (starL_l  -  end _ 1)  -f  time_int. duration  ; 

end 

frame  COMPRESS  is_a  LENGTH_CHANGE  with 
prerequisites 

subj  :  LONGITUDINAL_AXIS  ; 
start_l  :  such  that 

[  start_l  >  end_l  ]; 
dependents 

subj  :  OBJECT  with 

subj  <-  super_part(subj)  ; 
speed  :  SPEED_V  with 

speed  «-  (start_l  -  end _ 1)  -f  time_int. duration  ; 

end 

frame  LENGTHEN  is_a  LENGTH_CHANGE  with 
prerequisites 

subj  :  L0NGITUD1NAL_AX1S  ; 
start__l  :  such  that 

[  start_l  <  end _ 1  ]; 

dependents 

subj  :  OBJECT  with  subj  «-  super_part(subj)  ; 
speed  :  SPEED_V  with 

speed  *-  (end__l  -  start_l)  A  time_int. duration  ; 

end 

Different  types  of  LOCATION_CHANGES  are  built  up  from  the  TRANSLATE  and  ROTATE 

frames: 

frame  TRANSLATE  is_a  LOCATION.. CHANGE  with 

prerequisit  es 
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end 


subj  :  such  that 

[  for  ob2  :  CLOSE D_LINE_SEG  where 
[  ob2. centroid  =  subj  ]]; 

dependents 

subj  :  OBJECT  with  subj  <-  ob2  ; 
speed  :  SPEED_V  with 

speed  <-  (((end_loc.y  -  start_loc.y)  **  2  + 

(end _ loc.x  -  start_loc.x)  **  2)  **  0.5)  -f  time_int.  duration  ; 

traj  :  TRAJ_V  with 

traj  <-  arctan((end_loc.y  -  start_loc.y)  -r 

(end _ loc.x  -  start_loc.x))  ; 


frame  ROTATE  is_a  L0CATI0N_ CHANGE  with 
prerequisites 

centroid_mot  :  N0N_M0TI0N  such  that 

[  centroid_mot.subj  =  obi. centroid  ]; 
pl_mot  :  L0CAT10N_CHANGE  such  that 

[  pl_mot.subj  =  obl.long_axis.start_pt  ]; 

p2 _ mot  :  LOCATION_CHANGE  such  that 

[  p2_mot.subj  =  obl.long_axis.end _ pt  , 

(pl_mot.end _ loc.y  -  p  l_mot.start_loc.y)  -r 

(pl_mot.end _ loc.x  -  pl_mot.start_loc.x)  = 

(p2_mot.end _ loc.y  -  p2_mot.start_loc.y)  — 

(p2_mot.end _ loc.x  -  p2_mot.start_loc.x)  , 

length(pl_mot.start_loc,p l_mot.end _ loc)  -r 

length(p2_mot.start_loc,p2_mot.end _ loc)  = 

length(pl_mot.end _ loc, obi. centroid)  -r 

length(p2_mot.end_loc,ob  1. centroid)  ]; 

dependents 

angle  :  ANGLE_V  with 

angle  *-  ob l.long_axis. orient  at  pl_mot.end _ time  - 

obi. long_axis. orient  at  pl_mot.start_time  ; 
subj  :  OBJECT  with  subj  <-  obi  ; 

end 


The  motion  type  NON_MOTION  means  that  the  subj  is  stationary  for  the 

specified  time  interval.  It  is  further  explained  in  a  subsequent  section.  Finally, 

SKAPE_CKANGE  is  abstracted  to  distortions  such  as  : 

frame  STRAIGHTEN  is _ a  SHAPE_CHANGE  with 

prerequisites 

start_shape  :  OPEN_LINE_SEG  such  that 

[  start_shape  ~instance_of  SMOOTH _ LINE  ]  ; 

end _ shape  :  SMOOTH_LINE  ; 

end 

frame  BUCKLE  is_ a  SHAPE_CHAXGE  with 
prerequisites 

start_shape  :  SMOOTK_LINE  ; 
end _ shape  :  BUCKLED_LINE  ; 

end 

frame  DISTORT  is_a  SHAPE_CHANGE  with 
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end 


prerequisites 

subj  :  such  that 

[  subj.lat_axis.intersect_pt  at  time_int. start_time  ~same_as 
subj.lat_axis.intersect_pt  at  time_int.end _ time  ]  ; 


frame  WARP  is„a  SHAP EXCHANGE  -with 
prerequisites 

start_shape  :  SMOOTH_LINE  ; 

end _ shape  :  SHAPE  such  that 

[  end _ shape  ~instance_of  BUCKLED_LINE  and 

end _ shape  ~instance_of  SMOOTH_LINE  ]; 

end 


A  line  segment  with  a  buckle  is  a  smooth  curve  with  one  interior  bump  caus¬ 
ing  an  irregularity.  Therefore,  the  STRAIGHTEN  frame  defines  a  smoothing  out  of  this 
bump  so  that  the  resultant  line  segment  is  smooth  (not  necessarily  straight).  BUCKLE 
defines  the  opposite  shape  change.  WARP  is  the  catch-all  frame  for  any  other  more 
complicated  changes  in  the  shape  of  a  curved  segment.  DISTORT  is  used  to  describe 
changes  in  the  relative  positions  of  the  object’s  axes.  In  most  of  the  frames  given 
above,  the  IS_A  relationships  that  they  have  are  also  given.  This  defines  the  IS_A  rela¬ 
tionships  between  the  defined  frame  and  other  frames  in  the  knowledge  base.  The  pro¬ 
perties  of  IS _ A  were  discussed  in  the  representation  section.  The  next  section  will 

describe  how  IS_J\  can  be  used  to  construct  a  knowledge  base  of  motion  concepts. 


4.2  The  Organization  of  Motion  Concepts 


The  motion  concepts  described  thus  far  can  be  organized  using  the  IS _ A  rela¬ 

tionship  as  noted  above.  The  IS_A  relationship  is  a  partial  order  relation  and  defines  a 
hierarchy  along  which  frames  can  be  organized.  IS_A  also  implies  inheritance  of  pro¬ 
perties  from  father  to  son  nodes  in  the  knowledge  base.  All  slots  in  frame  A  ,  if  A  is  the 
father  of  j B  (  B  IS_A  j4  ),  are  inherited  by  B.  Frame  B  can  refer  to  any  slot  of  A  simply 
by  its  slot  name.  Extra  constraints  may  be  placed  on  the  slot,  or  other  slots  may  be 
added  in  frame  B  as  long  as  the  IS_A  relationship  is  still  semantically  valid.  The 
hierarchy  will  in  general  be  several  levels  deep. 

The  hierarchy  we  have  defined  for  motion  concepts  is  shown  in  Figure  4.1. 
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FIGURE  4.1  GENERAL  MOTION  IS_A  HIERARCHY 


The  left  side  of  the  hierarchy  describes  motion  concepts  relevant  to  single  object 
movements  with  respect  to  a  fixed  point  of  reference  (e.g.,  object  rotation).  Single  ob¬ 
jects  are  assumed  to  be  indivisible  and  may  represent  an  entity  or  part  of  an  entity. 
Simple  motions  are  further  classified  into  changes  of  location  (LOCATION_CHANGE)  or 
changes  of  physical  properties  (PHYS__PROP_CHANGE).  Here,  physical  property 
changes  are  changes  in  observable  object  attributes  such  as  length  or  area.  Since  all 
motions  are  in  two  dimensions  only  for  our  system,  such  changes  can  only  be  due  to 
motions  of  the  parts  of  the  object,  such  as  contractions.  Examples  are  shown  for  each 
of  these  classes  as  well.  The  right  hand  side  of  the  hierarchy  shown  in  Fig.  4.1  deals 
with  aggregate  motions.  Such  motions  are  further  classified  into  motions  of  objects 
defined  in  terms  of  the  motions  of  their  parts,  (SIMUL_MOT_PARTS),  super-imposed 
motions  for  a  single  object,  (SIMUL_MOT),  motions  involving  independent,  simultane¬ 
ously  moving  participants,  (SIMUL_DISTINCT_PARTS),  and  sequences  of  motions  for  a 
single  object  (SEQUENCE).  The  category  SIMUL_DISTINCT_PARTS  includes  the  motion 
classes  proposed  by  [Miller72],  This  classification  of  motion  concepts  is  based  on  work 
described  in  [Tsotsos76]. 

The  IS__A  hierarchy  shown  in  the  figure  will  now  be  described  in  detail.  Since 
this  is  a  hierarchy  for  motion  concepts,  it  is  fitting  that  the  root  node  be  called  "mo¬ 
tion".  The  MOTION  frame  is  : 

frame  MOTION  with 
prerequisites 

subj  :  OBJECT  ; 

time__int  :  TIME_INTERVAL  ; 

end 

Similarly,  there  is  a  NON_MOTION  frame,  which  is  instantiated  whenever  some 
subject  does  not  move  for  a  given  time  interval.  Its  form  is  exactly  as  that  for  MOTION, 
with  the  difference  being  the  name  change.  The  IS_A  descendents  of  MOTION  are  "sim¬ 
ple  motion"  and  "aggregate  motion".  These  frames  are  both  null  and  are  intended  for 
structuring  purposes  only.  Simple  motion  concepts  are  those  which  define  motions 
where  only  a  single  object  is  moving.  This  single  object  may  be  a  centroid,  i.e.,  transla- 
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tion  of  the  whole  single  object,  an  axis,  i.e.,  orientation  changes  of  a  whole  single  ob¬ 
ject,  indivisible  parts  of  an  object  (in  which  case  the  definition  is  recursive),  or  a  group 
of  separate  objects  moving  in  unison  such  as  an  airplane  configuration  (movement  of 
centroid,  or  axis  size  and  orientation  changes).  Keeping  this  in  mind,  the  simple  mo¬ 
tion  node  has  descendents  "LOCATION_CHANGE”  and  "PHYS_PROP_CHANGE”  (physical 
property  change).  The  definition  of  the  types  of  objects  which  can  exhibit  these  mo¬ 
tions  place  constraints  on  the  subjects  of  these  motion  concepts. 

Location  changes  are  strict  centroid  translations  or  strict  axis  orientation 
changes.  These  frames  (TRANSLATE  and  ROTATE)  have  already  been  given.  The 
TRANSLATE  frame  adds  in  the  "speed"  and  "traj”  slots  to  the  L0CAT10N_CHANGE  frame. 
It  would  thus  be  expected  that  constraints  on  these  slots  produce  the  concepts  below 
TRANSLATE.  Some  of  the  more  usual  concepts  are  those  defining  particular  trajec¬ 
tories  such  as  upwards  (the  verb  "rise"),  downwards  (the  verb  "fall"),  towards  (the  verb 
"approach"),  away_from  (the  verb  "recede”)  leftwards  and  rightwards.  The  frames  for 
each  of  these  follow,  with  constraints  placed  on  the  trajectory  slot  inherited  from  the 
TRANSLATE  frame.  Note  that  some  are  reflexive  (there  is  no  motion  referent  and  tra¬ 
jectory  values  are  numerical)  while  others  are  objective,  (there  is  a  motion  referent 
and  the  trajectory  comparisons  are  made  with  respect  to  the  position  of  this  referent). 

frame  REFL_TRANSLATE  is_a  TRANSLATE  with 
end 

frame  LEFTWARDS  is_a  RE FL_TRAN SLATE  Tirith 
prerequisit  es 

traj  :  such  that 

[  traj  >  5*7t/8  and 
traj  <  11*7t/8  ]; 

end 

frame  UPWARDS,  RISE  is_a  REFL_TRAN SLATE  with 
prerequisites 

traj  :  such  that 

[  traj  ^  rr/8  and 
traj  ^  7*tt/8  j; 

end 

frame  DOWNWARDS,  FALL  is_ a  REFL_TRAN SLATE  with 
prerequisites 

traj  :  such  that 
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[  traj  ^  9*7t/8  and 
traj  ^  15*7t/8  ]; 

end 

frame  RIGHTWARDS  is_a  RE FL_TRAN S LATE  -with 
prerequisit  es 

traj  :  such  that 

[  (  traj  ^  0  and 
traj  <  3*rr/8  )  or 
(  traj  ^  2*tt  and 
traj  >  13*7t/8  )  ]; 

end 


frame  OB J_TRAN SLATE  is_a  TRANSLATE  with 
prerequisites 

ref  :  CLOSED_LINE_SEG  ;  end 


frame  TOWARDS  ,  APPROACH  is _ a  OBJ _ TRANSLATE  with 

prerequisites 

traj  :  such  that 

[  (  traj  -  arctan((subj. centroid. y  -  ref. centroid. y)  4- 

(subj. centroid. x  -  ref.centroid.x)))  -  traj  ^  0.1  , 
length(subj. centroid,  ref.centroid)  at  time_int.start_time  > 

length(subj. centroid  ,  ref.centroid)  at  time_int.end _ time 


end 


]; 


frame  AWAY_FR0M  ,  RECEDE  is_a  0BJ_TRAN SLATE  with 
prerequisites 

traj  :  such  that 

[  (  traj  -  arctan((subj. centroid. y  -  ref. centroid. y) 

-f  (subj. centroid. x  -  ref.centroid.x)))  4-  (traj)  ^  0.1  , 
iength(subj. centroid  ,  ref.centroid)  at  time_int.start_time  < 

length(subj. centroid  ,  ref.centroid)  at  time_int.end _ time  ]; 


end 


The  frame  REFL_TRAN SLATE  is  empty  (no  slots),  while  0BJ_TRANSLATE  adds  a 
slot  for  a  "referent”  object  to  those  it  inherits  from  TRANSLATE.  Strict  rotation  may  be 
defined  as  being  either  clockwise  or  counterclockwise.  One  example  of  a  more  complex 
rotation  is  the  verb  "sway".  "Sway"  is  a  pendulum  like  motion,  with  one  end  of  the  pen¬ 
dulum  not  necessarily  fixed.  Therefore,  it  is  a  combination  of  a  strict  rotation  and  a 
strict  translation,  because  the  centroid  changes  location  and  the  object  changes 
orientation.  This  example  illustrates  another  feature  of  the  knowledge  base  -  super¬ 
imposed  motions,  which  will  be  discussed  shortly. 

PHYS_PROP_CHANGE  has  three  sub-classes  -  all  of  which  have  been  defined 

previously.  SHAPE_CHANGE,  AREA _ CHANGE,  and  LENGTH_CHANGE.  Several  examples 

of  elements  of  these  classes  were  also  presented  above.  Here  another  example  of 
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super-imposed  motions  can  be  found  in  the  verb  "extend"  which  is  a  combination  of 


"expand"  and  "lengthen". 

The  right  side  of  the  hierarchy  defines  combinations  of  motions:  several 
simultaneous  motions  for  a  single  object,  (sway,  extend,  walk  ),  simultaneous  motions 
for  each  of  an  object’s  parts,  sequences  of  motions  for  a  single  object  (contract,  ex¬ 
pand,  walk,  beat),  and  motions  with  distinct  participants  (lift,  pull,  or  any  motion  with 
two  or  more  unrelated  objects). 

Simultaneous  motions  for  a  single  object  are  quite  easily  defined  for  any 
number  of  motions.  The  motions  are  each  defined  by  a  slot  instance.  If  two  motions 
are  simultaneous,  then  the  instance  of  the  TIME_INTERVAL  frame  which  they  each 
refer  to  will  be  identical.  If  the  motions  overlap,  eg.,  a  "lift"  motion  followed  by  a  "lift 
while  translating  rightwards"  followed  by  a  "translate  righwards",  then  the  time  inter¬ 
vals  for  each  overlap.  Note,  the  verb  "lift"  is  a  propellant  type  motion  and  thus  has  an 
agent  and  an  object  slot.  In  general,  the  motions  included  in  a  frame  which  have  the 
same  subject  and  which  refer  to  the  same  instant  in  time,  (within  their  respective 
TIME_INTERVAL  instances),  are  super-imposed.  Examples  of  such  motions  are  the 
verbs  "extend"  and  the  example  of  lift  overlapping  with  a  rightwards  translation.  The 
frames  for  these  motions  follow: 

frame  EXTEND  is_ a  EXPAND,  LENGTHEN,  SIMUL_MOT  with 
prerequisites 

mot_l  :  EXPAND  ; 

mot_2  :  LENGTHEN  such  that 

[  mot_2.subj  =  mot_l.subj  , 
mot_l.time_int  =  mot_2.time_int  ]; 

end 

frame  LIFT__OVERLAP_RIGHTWARDS  is_a  SIMUL_MOT  with 
prerequisites 

mot_l  :  LIFT  ; 

mot_2  :  RIGHTWARDS  such  that 
[  mot_2.subj  =  mot_l.subj  , 

mot_2.time_int.start_time  <  mot_l.time_int.end_time  , 
mot_2.time_int.end_time  >  mot_l.time_int.end _ time  ]; 

dependents 

time_int  :  with 

start-time  <-  mot_l.time_int.start_time  , 

end _ time  «-  mot_2.time_int.end _ time  ; 

agent  :  with 
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end 


agent  «-  mot_l. agent  ; 
obj  :  "with 

obj  «-  mot _ l.obj  ; 

The  remaining  important  consideration  is  the  order  of  the  superposition.  Ro¬ 
tation  about  a  centroid  and  translation  of  a  centroid  are  clearly  commutative  and 

therefore  any  IS _ A  son  of  ROTATE  is  commutative  with  any  IS _ A  son  of  TRANSLATE. 

Shape  changes  are  defined  independently  of  object  location  and  orientation.  Therefore 
rotation  and  shape  change  are  commutative,  and  similarly,  translation  and  shape 
change  are  commutative.  In  addition,  there  is  no  change  in  centroid  location  for  shape 
changes,  so  shape  change  and  translation  are  commutative.  Since  shape  changes  may 
entail  axis  length  changes,  axis  length  changes  and  centroid  translations  are  commuta¬ 
tive.  Therefore  it  seems  to  be  true  that  motions  for  a  single  object  may  be  applied  to  it 
in  any  order  without  affecting  the  resultant  motion.  The  crucial  assumptions  here  are 
that  centroid  location  only  changes  as  a  result  of  its  translation,  and  that  the  axes  of 
the  object  always  pass  through  its  centroid.  These  are  reasonable  assumptions  be¬ 
cause  we  also  want  the  axes  and  centroid  to  have  semantic  significance,  so  they  should 
be  defined  with  respect  to  the  object  type. 

Sequences  are  exactly  what  the  name  implies.  A  particular  motion  sequence 

for  an  object  may  be  specified  by  including  as  a  part  of  the  sequence  frame  each  of  the 

components  of  the  sequence  and  linking  them  with  the  appropriate  timing  constraints. 

For  example  a  BEAT  is  a  contraction  followed  by  an  expansion.  This  example  follows. 

frame  BEAT  is_a  SEQUENCE  with 
prerequisites 

mot_l  :  CONTRACT  ; 
mot_2  :  EXPAND  such  that 

[  mot_2.subj  =  mot_l.subj  , 

mot_l.time_int.end_time  =  mot_2.time_int.start_time  ]; 
dependents 

time_int  :  with 

start_time  *-  mot_l.time_int.start__time  , 
end _ time  «-  mot_2.time_int.end _ time  ; 

end 

Motions  of  an  object  as  a  result  of  the  motions  of  its  parts  are  another  type  of 
simultaneous  motion.  A  motion  such  as  "walking”  is  too  complex  to  describe  in  terms 
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of  the  orientation  and  location  changes  of  each  leg  and  arm  and  torso  and  head.  In¬ 
stead,  the  motion  of  a  leg  or  arm  is  described  individually  (under  "simple  motion"). 


This  motion  would  be  a  specific  translation  and  rotation.  If  the  motions  for  each  part 
are  placed  in  a  single  frame,  then  their  resultant  would  be  the  motion  of  a  human  walk¬ 
ing  in  place  with  no  forward  translation.  This  frame  belongs  to  the  current  category. 
In  order  to  get  walking  motion  with  displacement,  the  "walk  in  place"  frame  is  placed  in 
another  frame  along  with  a  translate  instance  and  the  "walk"  frame  is  placed  under  the 


simultaneous  motion  node  in  the  hierarchy.  The  example  of  WALK  is  given. 


frame  WALK  is_a  SIMUL—MOT  with 
prerequisites 

mot_l  :  WALK_IN_PLACE  ; 
mot_2  :  TRANSLATE  such  that 

[  mot_2.time_int  =  mot_l.time_int  , 
mot_2.subj  =  mot_l.subj  ]; 
dependents 

time_int  :  urith 

time_int  *-  mot_l. time_int  ; 


end 


frame  WTALK_IN_PLACE  is_a  SEQUENCE  with 
prerequisites 

subj  :  PERSON  such  that 

[  mot_l.subj  part_of  subj  , 
mot_2.subj  part_of  subj  , 
for  ml  :  NON_MOTION  where 

[  ml. subj  =  self. subj. centroid  , 
ml.time_int  -  self.time_int  ]]; 
mot_l  :  RIGHT_LEG_FORWARD_SWING  ; 
mot_2  :  LEFT_LEG_FORWARD_SWING  such  that 

[  mot_l.time _ int.end _ time  =  mot_2.time _ int.start_time  ]; 

dependents 

time_int  :  with 

start_time  <-  mot_l.time_int.start_time  , 
end _ time  <-  mot_2.time_int.end _ time  ; 

end 


frame  RI GHT_LE G_F 0 RW ARD_  S  W1 N G  is_a  SIMUL_MOT_PARTS  with 
prerequisites 

subj  :  PERSON  such  that 

[  mot_l.subj  part_of  subj  , 
mot_2.subj  part_of  subj  , 
mot_3.subj  part_of  subj  , 
mot_4.subj  part_of  subj  ]; 
mot_l  :  RIGKT_LEG_FORWARD  ; 
mot_2  :  LE FT_ ARM_F 0 RW ARD  such  that 

[  mot_l.time_int  =  mot_2.time_int  ]; 
mot_3  :  LEFT_FOOT_ON_GROUND  ; 
mot_4  :  R1GKT_ARM_BACKWARD  such  that 
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end 


[  mot_4.time_int  =  mot_2.time_int  , 
mot_4.time  =  mot_l.time_int  ]; 
dependents 

time_int  :  with 

time_int  «-  mot_l.time_int  ; 


frame  LEFT_LEG_FORWARD_SWING  is_a  SIMUL_MOT_PARTS  with 
prerequisites 

subj  :  PERSON  such  that 

[  mot_l.subj  parL_of  subj  , 
mot_2.subj  part_of  subj  , 
mot_3.subj  part_of  subj  , 
mot_4.subj  part_of  subj  ]; 
mot_l  :  LEFT_LEG_FORWARD  ; 
mot_2  :  RI GHT_ ARM_F 0 RW ARD  such  that 
[  mot_l.time_int  =  mot_2.time_int  ]; 
mot_3  :  RIGHT_FOOT_ON_GROUND  ; 

mot_4  :  LEFT_ARM _ BACKWARD  such  that 

[  mot_4.time_int  =  mot_2.time_int  , 
mot_4.time_int  =  mot_l.time_int  ]; 
dependents 

time_int  :  with 

*-  mot_l.time_int  ; 

end 

Finally,  the  fourth  kind  of  simultaneous  motion  in  the  KB  is  that  for  simul¬ 
taneous  motions  but  for  distinct  participants.  They  are  motions  such  as  lifting  or  pul¬ 
ling.  The  motion  for  each  participant  may  be  described  by  some  other  frame  anywhere 
else  in  the  KB.  The  motions  are  then  placed  into  the  same  frame  along  with  any  extra 
constraints  as  appropriate.  For  example,  a  pulling  motion  requires  that  each  partici¬ 
pant  be  translating.  The  additional  constraints  are  that  the  objects  be  connected  in 
some  way,  and  that  their  trajectories  be  the  same,  and  that  they  occur  in  the  same 
time  interval.  The  frame  for  PULL  follows: 

frame  PULL  zs_a  PROPELLANT  with 
prerequisites 

mot_l  :  TRANSLATE  ; 

mot_2  :  TRANSLATE  such  that 

[  mot_2.subj  ~=mot_ l.subj  , 
mot_2.subj  behind  mot_l.subj  , 

mot_2.subj  connected _ to  mot_l.subj  , 

mot_2.traj  same.as  mot_l.traj  , 
mot_2.time_int  =  mot_l.time_int  ];  ; 
dependents 

time_int  :  with 

time_int  «-  mot_l.time_int  ; 
agent :  OBJECT  with 

agent  <-  mot_l.subj  ; 
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end 


obj  :  OBJECT  with 

obj  <-  mot_2.subj  ; 

The  assignments  for  the  dependent  slots  "agent"  and  "obj"  are  token  assign¬ 
ments  in  which  all  slots  of  the  "mot_l.subj”  token,  for  example,  are  copied  into  the 
corresponding  slots  of  the  OBJECT  token  for  the  "agent"  slot.  Clearly,  the  assignment 
is  meaningless  if  the  slot  names  do  not  correspond  exactly.  Frames  such  as  PROPEL¬ 
LANT,  SIMUL—MOT,  SIMUL_DISTINCT_PARTS,  etc.,  are  empty  and  are  used  for  organiza¬ 
tional  purposes  only. 

One  remaining  topic  which  must  be  described  in  this  chapter  is  the  various 
exception  types  that  are  relevant  for  motion  concepts.  Let  us  begin  with  those  for 

static  concepts.  LENGTH, _ V’s  can  be  either  TOO_SHORT  or  TOO _ LONG  with  the  subject 

of  the  exception  being  the  object  which  suffers  from  the  problem  and  the  referent  be¬ 
ing  the  particular  axis  type.  AREA _ V’s  can  be  TOO_LARGE  or  T00_SMALL  and  the  sub¬ 

ject  slot  is  filled  with  the  object  whose  area  has  the  wrong  value.  Other  types  of  rela¬ 
tions  that  can  produce  exceptions  on  mis-matches  are  the  spatial  relations  (such  as 

connectedness,  relative  locations,  etc.),  PART_OF  and  IS _ A.  Errors  in  spatial  relations 

produce  exception  types  of  the  form  NOT_spatial_relation,  with  the  subj  and  ref  slots 
filled  appropriately,  PART__OF  errors  produce  NOT_PART_OF  with  the  subj  filled  with  A 
and  the  ref  with  B  if  the  relation  is  A  PART_OF  B,  and  similarly  for  IS _ A.  The  excep¬ 

tions  arising  from  motion  anomalies  depend  on  which  of  the  motion  slots  are  involved. 
If  the  "speed"  slot  fails,  the  subject  may  be  moving  TOO_FAST  or  T00_SL0W,  either  re¬ 
lative  to  a  numerical  velocity  or  to  another  object.  In  the  latter  case  a  referent  slot 
would  be  bound  to  that  other  object,  and  in  both  cases  the  subject  slot  is  filled  with  the 
motion  subject.  Mis-matches  in  trajectories  are  handled  by  WRONG__TRAJ  and  its  slots 
are  the  same  as  for  "speed"  failures.  There  are  many  types  of  exceptions  for  timing  er¬ 
rors.  A  motion  may  have  TOO_LONG_DUR,  too  long  a  duration  or  TOO_SHORT_DUR.  A 
motion  may  be  EARLY  or  LATE,  both  relative  to  a  numerical  time  value  or  to  another 
motion.  Also,  a  motion  may  NOT_NEXT,  not  be  the  next  motion  as  specified  in  a  frame 
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or  may  not  be  simultaneous  with  another  motion,  NOT_SAME_TIME. 

All  of  the  above  mentioned  exception  types  are  automatically  raised  in  the 
matching  procedure  simply  because  they  are  associated  with  the  standard  slot  names. 
Exceptions  for  user  slot  names  must  be  explicitly  defined  within  a  frame,  and  the  se¬ 
mantics  of  the  exception,  i.e.,  its  slots,  specified. 


CHAPTER  5 


LEFT  VENTRICULAR  WALL  MOTION 
CONCEPTS 


5.1  The  Problem  Domain 

Left  ventricular  angiography  is  a  widely  used  cardiac  imaging  technique. 
Briefly,  the  procedure  is  as  follows.  A  catheter,  which  is  a  long  thin  plastic  tube,  whose 
end  is  shaped  appropriately  to  facilitate  its  use,  is  inserted  into  either  the  bracchial 
(upper  arm)  or  femoral  (upper  thigh)  artery.  It  is  then  pushed,  turned  and  twisted  by 
skilled  hands  until  it  reaches  the  desired  location  of  the  heart.  It  can  be  inserted  into 
any  of  the  coronary  arteries  via  the  aorta,  into  any  chamber  of  the  heart,  or  into  the 
pulmonary  system.  To  reach  the  left  ventricle  (LV),  the  catheter  is  most  commonly  in¬ 
serted  through  the  aorta  and  into  the  cavity.  Once  in  the  desired  location,  a  radio¬ 
paque  dye  is  injected  through  the  catheter.  This  dye  mixes  with  the  blood,  thus  creat¬ 
ing  on  the  X-ray  image  light  areas  (more  X-rays  absorbed)  wherever  the  dyed  blood  is 
present.  Figure  5.1  shows  such  an  actual  image  with  the  corresponding  left  ventricu¬ 
lar  epicardial  outline.  The  central  blob  is  the  left  ventricle.  The  aorta  is  the  structure 
at  the  top  of  the  LV.  On  the  left  upper  side,  the  outline  is  quite  sharp  -  this  is  the  loca¬ 
tion  of  the  mitral  valve.  Traversing  counterclockwise  around  the  LV,  the  outline  be¬ 
comes  more  blurred.  Actually,  the  fuzziness  is  due  to  the  fact  that  not  only  the  inner 
wall  (endocardium)  is  visible  but  also  the  outer  wall  (epicardium)  and  they  are  quite 
easily  separated  by  eye.  At  the  lower  left,  the  top  portion  of  the  diaphragm  is  visible. 
Figure  5.2  shows  an  LV  outline  with  various  features  marked.  This  figure  should  be  re- 


-  65  - 


FIGURE  5.1 


LV  IMAGE  AND  TRACED  EPICARDIAL  OUTLINE 
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FIGURE  -5,2 


THE  LEFT  VENTRICLE  AND  ITS  PARTS 
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ferred  to  when  necessary  during  the  remainder  of  this  document. 

A  film  is  taken  in  this  manner  with  60  images  per  second  or  roughly  50  images 
per  beat  for  a  normal  heart.  In  addition,  radiopaque  crosshairs  may  be  placed  on  the 
patient’s  chest  during  the  procedure  thus  providing  an  absolute  point  of  reference  for 
the  motion  observed.  Synchronized  ECG  and  pressure  may  be  taken  with  the  film.  To 
record  pressure,  a  second  catheter  is  required,  equipped  with  a  pressure  transducer. 

This  imaging  technique  has  several  inherent  problems: 

1)  Because  the  X-rays  come  from  a  point  source  and  the  exact  distance  from  the 
source  to  the  LV  is  uncertain,  corrections  for  magnification  and  distortion  of  the  image 
must  be  determined  and  applied  before  further  processing. 

2)  The  image  actually  seen  does  not  necessarily  correspond  to  a  plane  through 
the  left  ventricle  which  is  perpendicular  to  the  axis  of  the  X-ray  beam. 

3)  Dye  mixing  is  often  incomplete  and  the  opacified  blood  does  not  completely  fill 
the  cavity.  Thus,  the  boundary  of  the  light  areas  may  not  correspond  exactly  to  the 
cavity  outline.  In  addition,  artifacts  of  the  outline  are  caused  by  the  papillary  muscles, 
the  trabeculae  (folds  of  tissue)  on  the  endocardium  and  by  the  cusps  of  the  mitral 
valve.  If  any  of  these  parts  of  the  LV  either  trap  blood  or  become  prominent,  they 
create  misleading  images.  There  are  also  other  objects  which  can  obstruct  the  image, 
namely  the  ribs,  sternum,  spine,  the  catheter  itself  and  the  diaphragm.  These  must  be 
removed  or  bypassed  in  some  way  for  proper  analysis. 

4)  In  any  one  view,  anomalies  of  motion  may  be  missed,  if  the  axis  of  the  abnor¬ 
mal  region  is  not  relatively  perpendicular  to  the  axis  of  the  X-ray  beam.  Thus,  one 
must  be  aware  that  one  or  even  two  perpendicular  views  may  not  tell  the  whole  story. 

Magnification  effects  have  been  adequately  taken  care  of  by  medical 
researchers  and  this  is  documented  in  [Heintzen7l].  The  second  problem  can  be  dealt 
with  in  the  following  manner.  The  cardiologist  doing  the  angiogram  may,  in  a  trial  and 
error  manner,  find  the  best  compromise  angles  for  taking  the  film.  Secondly,  if  the 
internal  model  of  the  LV  is  three  dimensional,  a  projection  of  it  can  be  taken  in  the 
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same  direction  as  the  X-rays  irradiate  the  patient’s  chest.  This  projection  should  then 
correspond  to  the  actual  image.  A  two  dimensional  model  would  not  correspond  exact¬ 
ly  to  the  LV's  physical  characteristics;  but  nevertheless,  it  would  still  be  of  great  use  to 
clinicians.  We  will  begin  our  work  with  a  2D  model  and  leave  the  extension  to  the  3D 
case  for  future  research. 

As  for  the  dye  mixing  problem,  we  will  assume  that  only  images  in  which  the 
dye  mixing  is  acceptable  will  be  analyzed  by  the  system.  However,  artifacts  produced 
by  the  components  of  the  LV  will  be  handled,  and,  obstructing  objects  will  be  recog¬ 
nized  and  taken  into  consideration.  The  fourth  problem  cannot  be  dealt  with  unless  3D 
images  are  provided.  By  analysing  several  2D  views  it  may  be  possible  to  limit  the 
chance  for  error. 

An  additional  problem  arises  due  to  the  nature  of  left  ventricular  function. 
While  the  LV  contracts  and  expands,  the  forces  within  it  cause  it  to  move  about  or  rock 
within  the  chest  cavity.  This  is  visible  in  some  patients  as  a  distinct  pendulum  type 
motion  of  the  apex  of  the  ventricle.  In  other  patients,  it  is  not  visible  at  all. 

This  rocking  motion  may  be  recognized  and  accounted  for  in  the  following 
manner.  By  observation,  we  notice  that  the  aorta  moves  in  a  roughly  up  and  down 
manner,  such  that  the  centroid  of  the  sinus  of  valsalva  moves  along  a  corridor.  We  can 
also  use  the  fact  that,  because  of  the  shape  characterization,  the  apex  will  always  be 
chosen  in  a  consistent  manner.  Now,  if  a  line  joining  the  centroid  of  the  sinus  and  the 
apex  changes  slope  significantly,  we  know  that  the  LV  has  rocked.  In  addition,  the 
amount  of  rocking  motion  to  use  to  correct  the  apparent  motion  of  the  segments  to 
true  motion,  can  be  obtained  by  the  displacement  of  this  line  along  its  length  from  im¬ 
age  to  image.  Obviously,  this  correction  will  add  uncertainty  to  the  results,  because 
the  rocking  is  not  like  a  rigid  pendulum,  but  is  more  variable.  However,  the  result 
should  be  close,  and  will  definitely  be  much  more  correct  than  if  no  correcting  meas¬ 
ures  were  used  for  rocking  motion.  Fig.  5.3  illustrates  this. 

What  can  be  derived  from  these  films  and  how  is  the  analysis  done  by  the  car- 


-  67  - 


e 


h 


-  correction  ^  r 
point-,  or.  f 
corrector,  ir. 
to  polntr  on  r 

-  correction  '"or 
centrclc;  o'"  rir.us 

-  correction  Per 
apex 


FIGURE  5.3  ROCKING  MOTION  AND  CORRECTIONS 


67a 


diologists?  Well,  the  answer  to  the  second  question  is  easy.  A  cardiologist,  together 
with  his  students,  watches  the  films  with  a  cinefilm  projector.  The  projector  has  for¬ 
ward,  reverse,  variable  speed  and  stop  frame  features.  Several  years  of  experience  en¬ 
ables  them  to  determine  the  types  of  wall  motion  exhibited  by  each  portion  of  the  wall. 
Clearly  this  is  not  always  consistently  and  objectively  done.  The  type  of  learning  which 
the  cardiology  resident  experiences  is  "learn  by  example"  or  "abstract  from  a  set  of 
example  types”.  Thus,  the  abstracted  model  which  he  has  in  his  mind  is  very  difficult 
for  him  to  express.  It  was  necessary  to  keep  this  in  mind  while  we  were  learning  about 
LV  motion  so  that  we  could  put  our  abstractions  down  on  paper.  Such  knowledge  and 
models  cannot  be  found  in  any  cardiology  text. 

Figure  5.4  shows  the  form  filled  out  by  the  cardiologist  summarizing  his 
findings  about  a  film.  There  are  also  two  other  basic  measurements  taken  which  are 
commonly  used  in  diagnosis:  LV  volume,  both  end-diastolic  (largest)  and  end-systolic 
(smallest),  and  ejection  fraction,  derived  from  the  volumes.  Only  two  images  are  rou¬ 
tinely  measured  due  to  the  amount  of  work  required  by  the  human  operator.  However, 
manual  tracing  of  certain  films  in  their  entirety  is  done  when  required  and  plots  of 
volume  vs.  time  are  produced.  Volume  is  calculated  based  on  various  models  using  the 
maximum  dimension,  area  and  other  such  measurements. 

Let  us  return  to  the  form  on  Figure  5.4  and  discuss  some  of  its  features.  One 
of  the  first  points  that  one  notices  is  that  there  are  two  views  of  the  LV  -  the  RAO,  right 
anterior  oblique,  and  the  LAO,  left  anterior  oblique.  These  terms  simply  describe  the 
direction  to  which  the  X-rays  are  beamed.  Other  views  are  possible  because  the  ap¬ 
paratus  used  can  rotate  fully  360  degrees  around  the  patient;  however,  these  two  are 
used  as  a  compromise  between  having  the  beam  perpendicular  to  the  axis  of  a  normal 
LV  and  avoiding  structures  such  as  the  ribs,  sternum  and  spine  which  would  obstruct 
the  LV.  Secondly,  notice  that  the  LV  outline  is  divided  into  5  regions  for  the  RAO  view 
and  3  for  the  LAO  view.  Both  views  show  the  anterobasal  region,  thus  giving  a  total  of  7 
regions.  The  apical,  diaphragmatic  and  septal  regions  are  related  to  actual  physical 
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components  -  the  apex,  the  diaphragm  and  the  septum.  The  others  simply  give  the 
general  location  of  the  region.  For  each  of  these  regions,  the  cardiologist  must  deter¬ 
mine  the  motion  class  which  it  exhibits.  There  are  six  classes  of  motion:  normal;  hy¬ 
pokinetic  -  the  segment  exhibits  less  than  normal  motion;  akinetic  -  the  segment  exhi¬ 
bits  no  motion;  dyskinetic  -  the  segment’s  motion  is  out  of  phase  with  respect  to  dias¬ 
tole  and  systole;  aneurysmal  -  the  segment  absorbs  energy  during  systole  and  releases 
it  during  diastole;  and,  undefined. 

Now,  let  us  discuss  the  problems  with  these  classes. 

1)  Any  abnormality  does  not  necessarily  fall  entirely  within  one  of  the  pre-defined 
segments.  Thus,  if  such  an  abnormality  crosses  borders,  it  is  not  true  that  all  portions 
of  both  regions  suffer  from  the  problem. 

2)  The  extent  of  the  regions  and  the  characteristics  of  the  abnormality  classes 
are  not  defined  in  a  precise  manner.  They  depend  on  the  viewer's  subjective  judge¬ 
ment. 

3)  The  heart  may  rock  in  the  chest  cavity  and  this  is  entirely  normal.  The  prob¬ 
lem  that  this  creates  is  that  there  is  superimposed  motion  on  the  regular  contraction- 
relaxation  pattern.  This  causes  problems  for  the  human  viewer,  as  he  must  separate 
the  two  in  order  to  get  an  objective  assessment  of  both  motions. 

4)  The  definitions  of  dyskinetic  and  aneurysmal  motions  give  rise  to  ambiguities 
for  they  both  seem  to  define  the  same  thing. 

5)  Akinetic  motion  can  occur  in  two  ways  -  but  not  in  the  manner  in  which  the 
definition  implies.  It  is  not  likely  that  a  segment  stays  completely  stationary  while  its 
adjacent  segments  move.  Physics  would  tell  us  that  the  moving  segments  would  pull 
the  inactive  segment  along  with  them.  Thus,  it  does  not  move  -  but  displays  an  abnor¬ 
mal  motion  pattern  -  a  "passive"  motion  pattern.  The  two  ways  in  which  inactive  seg¬ 
ments  reveal  themselves  is  either  by  exhibiting  no  or  little  shape  change  or  by  exhibit¬ 
ing  a  smellier  velocity  of  motion.  In  both  cases,  the  segment  would  lag  behind  the  mo¬ 
tion  of  its  neighbours. 
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6)  In  light  of  this  view  of  akinesis,  one  wonders  what  hypokinesis  should  mean? 

From  the  above  comments,  it  is  clear  that  certain  decisions  must  be  made 
concerning  the  LV  segments  and  motion  classes  which  our  system  will  consider.  These 
were  carefully  and  thoroughly  discussed  with  several  members  of  the  Cardiovascular 
Unit  at  Toronto  General  Hospital.  ALVEN,  at  first,  will  look  only  at  the  RAO  projection  of 
the  LV.  The  LV  wall  will  be  divided  into  3  segments:  posterior,  apical  and  anterior. 
These  segments  are  determined  in  relation  to  the  centroid  of  the  LV.  The  centroid  is 
defined  as  the  point  which  is  63%  of  the  way  down  from  the  aortic  valve  edge,  along  the 
line  joining  the  apex  and  the  aortic  valve  edge  in  the  end-systoiic  image  (smallest 
volume).  This  technique  is  taken  from  [Alderman79].  The  segments  are  then  derived 
by  dividing  the  angle  formed  by  the  mitral  valve  edge  point,  the  centroid  and  the  aortic 
valve  edge  (shown  in  Figure  5.2)  into  3  equal  segments.  The  longitudinal  axis  is  defined 
as  the  line  connecting  the  apex  and  the  aortic  valve  edge,  while  the  lateral  axis  is  the 
line  joining  the  mitral  valve  edge  and  intersecting  the  longitudinal  axis  at  a  right  angle. 

The  current  version  of  ALVEN  considers  only  3  motion  classes:  normal, 
ischemic  and  dyskinetic.  Ischemic  heart  disease  leads  to  hypokinetic  and  akinetic  mo¬ 
tions.  Dyskinetic  here  means  motions  which  are  too  fast  (hyperkinetic)  or  motions 
with  improper  trajectories  (paradoxical). 

5.2  Motivation 

Quantitative  assessment  of  the  functional  state  of  the  human  left  ventricle 
demands  the  acquisition  and  processing  of  data  relating  to  left  ventricular  (LV)  size, 
shape,  the  extent  and  velocity  of  wall  motion,  the  ejection  efficiency,  and  the  integra¬ 
tion  of  this  data  with  the  instantaneous  pressure  within  the  ventricular  cavity. 
Although  it  is  possible  to  confine  analysis  to  a  single  end-systolic  and  end-diastolic 
frame,  complete  LV  function  assessment  demands  more  detailed  study. 

Usually,  30  to  60  sequential  images  are  measured  from  a  cine  film  of  the  ven¬ 
tricular  cavity  during  a  single  cardiac  cycle  lasting  about  1  second.  These  measure- 
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ments  are  combined  with  a  similar  number  of  pressure  measurements.  Carrying  such 
analysis  out  manually  is  extremely  time-consuming,  tedious  and  subject  to  many  er¬ 
rors.  The  human  does  not  have  the  ability,  by  eye  alone,  to  select  border  points  which 
are  in  correspondence  throughout  the  image  sequence.  Thus,  border  determination  is 
a  subjective  and  arbitrary  process.  Measuring  a  single  cycle  in  such  a  manner  requires 
from  1  to  3  hours. 

Even  if  detailed  ventricular  function  studies  are  limited  to  cineangiographic 
records,  the  load  faced  is  approximately  1000  patients  per  year.  New  non-invasive 
techniques  such  as  2-dimensional  echocardiography,  nuclear  angiography  (or  gamma 
imaging),  and  video  flouroscopic  recording  of  tantalum  screw  markers  embedded  in 
the  ventricular  wall  during  cardiac  surgery,  have  increased  this  load  dramatically. 

A  variety  of  techniques  have  been  attempted  for  more  rapid  and  accurate 
processing  of  cardiac  images.  Automatic  systems  have  attempted  to  measure  all  data 
of  interest  in  each  frame  without  human  intervention  [Reiber75],  [Clayton74], 
[Chow72],  [Tasto74].  Viable  systems  are  still  not  available.  Semi-automatic  systems 
exist  wherein  a  human  operator  manually  outlines  the  ventricular  cavity,  using  a  digi¬ 
tizing  device  which  inputs  border  co-ordinates  to  a  computer  [Cowey72],  [Sturm72], 
[Heintzen74].  Measurements  are  done  on  the  digitized  border  typically  by  a  small 
computer.  Such  systems  are  limited  by  the  accuracy  and  speed  of  the  person  who 
does  the  outlining. 

Why  have  such  systems  failed?  It  appears  that  there  are  three  main  reasons: 

1)  They  attempt  to  extract  a  border  from  a  complex  image  by  incorrectly  imput¬ 
ing  to  the  border  some  gross  signal  characteristics.  They  subject  each  picture  point  to 
an  analysis  which  will  detect  the  presence  of  these  characteristics,  thus  defining  bord¬ 
er  points.  The  problem  is  that  such  characteristics  are  often  not  unique  to  the  border 
points.  When  the  computer  is  fooled,  it  is  fooled  permanently  and  the  human  must  in¬ 
tervene. 

2)  Even  if  an  automatic  border  detection  system  could  be  constructed  it  answers 
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only  part  of  the  problem:  that  of  producing  raw  border  data.  The  data  output  from 
ventricular  function  studies  is  voluminous  and  hard  to  understand.  The  result  is  that 
the  system  which  was  supposed  to  solve  the  problem  produces  output  which  itself 
causes  further  problems.  In  other  words,  the  system  cannot  abstract  the  essential 
data,  summarize  it,  or  distill  the  important  information  from  the  irrelevant. 

3)  Humans  do  not  have  the  ability  to  separate  complex  superimposed  motions 
from  each  other  by  eye  alone.  One  clear  example  of  this  is  the  problem  of  quantitative 
analysis  of  segmental  wall  motion  in  a  left  ventricle  which  rocks  and  exhibits  aneu¬ 
rysmal  behaviour.  Also,  current  systems  choose  to  follow  segmental  motion  according 
to  arbitrary  axes  ,  eg.,  radial  or  hemi-axial  motion. 

Such  problems  motivated  our  choice  of  cine  angiograms  as  the  problem 
domain  on  which  to  the  test  our  methodology  for  visual  motion  understanding. 


5.3  Left  Ventricular  Motion  Concepts 

As  mentioned  earlier,  the  outline  of  the  left  ventricle  will  be  described  by  par¬ 
titioning  the  outline  into  3  segments:  posterior,  apical  and  anterior  (when  viewed  in 
the  Right  Anterior  Oblique  projection).  These  segments  form  the  parts  of  the  LV’s  ob¬ 
ject  frame.  An  LV  is  a  CONTRACTILE__OBJECT,  while  a  N_LV  (normal  LV)  is  : 

frame  LV  is_a  CQNTRACTILE__OBJECT 

end 


frame  N_LV  is _ a  LV  -with 

prerequisites 

ant_seg  :  N_ANT_SEG  such  that 

[  ant_seg.start„pt  =  ap_seg.end_pt  , 
an t_s eg. end__.pt  =  long_axis.start_.pt  ]; 
ap_seg  :  N_AP_SEG  such  that 

[  ap_seg.start_pt  =  post_seg.end_.pt  , 
ap_seg.depth_pt  =  long_axis.end_pt  ]; 
post_seg  :  N_POST_SEG  such  that 

[  post_seg.start_pt  =  mv.end _ pt  ]; 

mv  :  MITRAL_VALVE  ; 
av  :  AORTIC_VALVE  ; 
dependents 

arclen  :  with 

arclen  <-  ant_seg. arclen  +  post„seg. arclen  + 
ap_seg. arclen  ; 


end 
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The  parts  of  an  LV  are  those  noted  in  Fig.  5.2  -  the  mitral  valve,  the  aortic 

valve,  and  the  three  segments,  apical,  anterior  and  posterior.  Since  the 

CONTRACTILE_OBJECT  frame  is  an  ancestor  of  the  LV  frame,  all  LV’s,  whether  normal 

or  not,  have  the  slots  area,  longitudinal  axis,  lateral  axis  and  arclength.  In  addition, 

each  of  the  three  segments  are  defined  by  three  points,  the  start,  end  and  mid-points 

(depth_pt).  The  frames  for  the  three  segments  are: 

frame  N_ANT_SE G  is_a  CURVED_SEG 
end, 

frame  N_AP_SEG  is_a  CURVED_SEG 
end 

frame  N_POST_SEG  is_a  CURVED_SEG 
end 

The  slots  "start_pt”,  ’’end _ pt”  and  "arclength”  are  inherited  from  the 

LINE_SEG  frame.  The  remaining  parts  of  the  LV  frame,  the  valves,  are  simply  line  seg¬ 
ments. 

LV  motions  (cycles),  both  normal  and  abnormal,  fall  under  sequence  motions 
in  the  IS_A  hierarchy.  (See  Fig.  5.5  for  the  frames  which  fall  below  this  node.  Note  that 
all  normal  LV  motions  are  pre-fixed  with  "N_",  while  all  abnormal  ones  are  pre-fixed 
with  "ABN_").  While  the  form  of  such  motions  is  not  known  precisely,  there  are  some 
subjective  data  for  structuring  an  initial  version  of  the  knowledge  base.  In  particular, 
there  are  7  distinct  phases  to  a  left  ventricular  beat  (assuming  no  arrhythmias) 
[Silber75],  for  which  approximate  timing  relationships  are  known.  Not  much  is  known 
about  absolute  velocities  and  trajectories  of  the  segments.  However,  relative  velocities 
and  trajectory  can  be  included.  An  important  component  of  ALVEN  is  a  data  collection 
system  whose  purpose  is  to  determine  this  information  more  precisely,  thus  allowing 
an  iterative  refinement  of  the  knowledge  base. 

The  left  ventricular  cycle  has  two  main  phases  -  systole  (ejection,  or,  contrac¬ 
tion)  and  diastole  (filling,  or,  expansion).  This  is  the  reason  for  placing  the  left  ventric¬ 
ular  cycle  frame  beneath  the  BEAT  frame  in  the  IS_A  hierarchy.  In  systole,  the  left 
ventricle  contracts  and  expels  blood  into  the  aorta,  while  in  diastole  it  expands  and  fills 
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FIGURE  5.5  NORMAL  LV  MOTION  IS  A  HIERARCHY 


with  blood.  The  systolic  phase  is  divided  into  three  parts:  the  isovolumic  contraction 
phase,  the  maximum  ejection  phase  and  the  reduced  ejection  phase,  while  diastole  is 
divided  into  four  phases:  isovolumic  relaxation,  rapid  filling,  diastasis  and  filling  by  atri¬ 
al  contraction. 

The  main  events  which  occur  during  each  phase  are  now  described.  The  aver¬ 
age  durations  of  each  phase  are  given  in  seconds  with  a  heart  rate  of  75  beats  per 
minute.  Event  durations  vary  with  the  heart  rate.  During  each  phase  events  are 
classified  by  whether  they  occur  at  the  beginning,  middle  or  end  of  the  phase.  Thus, 
the  descriptions  will  be  pre-fixed  by  B,  M,  or  E  respectively. 

Isovolumic  Contraction  -  duration  0.05 
B)  onset  of  ventricular  contraction 

M)  no  ventricular  volume  change,  but  there  are  shape  changes  of  the  cavity,  the  atrio¬ 
ventricular  valves  close  and  there  is  a  rapid  rise  in  ventricular  pressure. 

E)  opening  of  aortic  valve 

Maximum  Ejection  -  duration  0.09 
B)  opening  of  aortic  valve 

M)  rapid  outflow  of  blood  from  the  ventricle  to  the  aorta 
E)  intraventricular  pressure  reaches  its  peak 
Reduced  Ejection  -  duration  0.13 
B)  peak  of  ventricular  pressure 
M)  declining  flow  of  blood 
E)  onset  of  ventricular  relaxation 

Isovolumic  Relaxation  -  duration  0.12 
B)  onset  of  ventricular  relaxation 

M)  rapid  drop  of  ventricular  pressure,  aortic  valve  closes,  no  volume  change  but  there 
may  be  shape  changes 
E)  atrio-ventricular  valves  open 

Rapid  Filling  -  duration  0.11 
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B)  atrio-ventricular  valves  open 

M)  rapid  flow  of  blood  from  left  atrium  into  left  ventricle 
E)  slowing  of  flow  of  biood 

Diastasis  -  duration  0.19 
B)  slowing  of  blood  flow 
M)  continued  slower  flow 
E)  onset  of  atrial  contraction 

Filling  by  Atrial  Contraction  -  duration  0.11 

B)  onset  of  atrial  contraction 

M)  increased  flow  of  blood  into  left  ventricle  from  atrium 
E)  onset  of  ventricular  contraction 

Shape  changes  are  very  difficult  to  define.  Moreover,  they  are  not  at  all  well 
known  to  cardiologists.  They  will  be  dealt  with  in  terms  of  area  changes,  because  dur¬ 
ing  the  isovolumic  phases,  the  area  changes  are  small.  Pressure  and  valve  motions  are 
not  included  in  this  version  of  the  knowledge  base.  The  action  of  expulsion  of  blood  is 
due  to  a  contraction  of  the  ventricle,  while  filling  leads  to  expansion.  These  motions 
may  be  handled  in  two  independent  ways.  If  the  area  vs.  time  relationships  are  known 
for  each  ventricular  class,  explicit  contract  and  expand  events  may  be  used  to 
describe  them.  These  relationships  are  difficult  to  obtain  and  depend  heavily  on  the 
size  of  the  ventricle.  No  such  classification  exists  in  the  cardiology  literature.  The 
second  way,  and  the  one  which  will  appear  below,  is  that  during  normal  contraction, 
each  of  the  three  segments  move  inward  (towards  the  centroid)  with  relatively  the 
same  velocity.  Similarily,  during  expansion  they  move  outwards  (away  from  the  cen¬ 
troid)  with  similar  velocities. 

Using  such  information,  a  sketch  of  the  knowledge  base  can  be  constructed. 
There  is  a  separate  motion  frame  for  each  phase.  Some  phases  define  contraction  of 
the  LV  (maximum  ejection,  reduced  ejection),  others  expansion  (rapid  filling,  diastasis, 
filling  by  atrial  contraction),  and  two  define  a  state  of  iso-metric  fucntion  of  the  ventri- 
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cle  (isovolumic  contraction,  isovolumic  relaxation  -  isovolumic  is  the  key  here).  There¬ 
fore  the  phase  frames  are  IS _ A.  linked  to  the  appropriate  frames  for  "contract"  and 

"expand"  as  well  as  to  the  appropriate  aggregate  motion  frame.  In  each  frame,  all  of 
the  segments  of  the  LV  are  associated  with  the  event  or  sequence  of  events  that  define 
the  segment’s  motions  during  that  phase.  Correspondingly,  the  frame  for  a  normal  LV 
cycle  would  be  made  up  of  the  events  which  represent  each  phase  of  the  LV,  linked  in  a 
sequence  with  appropriate  timing  information  so  that  together  they  define  the  cycle. 
The  frame  for  the  normal  left  ventricular  cycle  is  given  below.  The  cycle  is  split  into 
two  main  phases,  systole  and  diastole.  The  decision  was  made  to  represent  "clinical" 
systole  rather  than  "hemodynamic"  systole  for  the  following  reasons.  Clinical  systole  is 
the  more  easily  definable  of  the  two  for  the  clinician  because  it  is  bounded  by  the  first 
and  second  heart  sounds  (closing  of  the  mitral  valve  and  the  closing  of  the  aortic 
valve).  Hemodynamic  systole,  on  the  other  hand,  is  the  time  interval  during  which  the 
mitral  valve  is  closed  and  thus  includes  the  isovolumic  relaxation  period,  and  is  harder 
to  define  clinically. 
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frame  N_LV„ CYCLE  is _ a  BEAT  with 

prerequisit  es 

subj  :  N__LV  ; 

systole  :  N_SYSTOLE  such  that 
[  systole. time_int.encl_time 

=  diastole. time_int.start_time  , 
systole. subj  =  self.subj  ]; 
diastole  :  N_D1AST0LE  such  that 
[  diastole. subj  =  self.subj  ]; 
dependents 

time_int  :  with 

start_time  «-  systole. time_int.start_time  , 

end _ time  <-  diastole.  time_int.end_time  , 

duration  default  0.8  *  AD  /  0.8  such  that 
[  duration  ^  0.98  *  AD  /  0.8  , 
duration  §  0.53  *  AD  /  0.8  ]; 
similarity  links 

sim _ linkl  :  ISCKEMIC_CAP_LV_CYCLE 

for  differences  : 

dl  :  N0_N_SY STOLE  where 

[  prereq_part  instance__of  ISCK_CAP„SYSTOLE  ], 
d2  :  N0_N_DIAST0LE  where 

[  prereq__part  instance_of  ISCH__CAP_DIASTOLE  ]; 
bound  by  : 

subj  -*  subj  ;; 

end 
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In  the  above  frame,  a  similarity  link  was  included  for  the  case  where  the  api¬ 
cal  segment  exhibited  abnormal  motion.  Note  that  NO_N_SYSTOLE  and 
NO_N_DIASTOLE  are  exception  frame  types  while  "prereq_part”  is  one  of  their  slots  as 
discussed  in  section  3.2.3.  They  correspond  to  the  differences  that  must  be  observed 
so  that  the  link  would  be  activated.  The  cases  for  the  anterior  or  posterior  segments 

would  have  precisely  the  same  form,  linking  the  ISCH _ CANT_LV_  CYCLE  and 

ISCH_CPOST_LV_CYCLE  frames  respectively,  with  the  appropriate  changes  to  the 
values  of  the  "subj"  slot  of  the  exception  and  to  the  bindings.  In  the  remainder  of  the 
frames,  only  that  abnormality  will  be  included;  however,  it  should  be  clear  that  the  oth¬ 
er  two  classes  are  handled  in  precisely  the  same  manner. 

AD  is  the  actual  duration  of  the  patient’s  heart  cycle  and  is  used  as  a  scaling 
factor.  The  above  frame  clearly  does  not  contain  all  the  information  that  is  necessary 
to  define  a  normal  LV  cycle.  Its  parts  are  further  decomposed  according  to  the  hierar¬ 
chy  shown  in  Fig.  5.6  and  frames  are  defined  for  each  of  those  concepts  as  well.  The 
frames  in  the  next  level  of  the  PART_OF  hierarchy  are  those  for  each  of  the  two  phases 
in  the  N_LV_CYCLE  frame  -  systole,  and  diastole. 
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N  LV  CYCLE 
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FIGURE  5.6  NORMAL  LV  CYCLE  PART_OF  HIERACHY 


frame  N_SYSTOLE  is_a  SEQUENCE  with, 
prerequisites 

subj  :  N_LV  ; 

isovol _ contract  :  N_ISOVOLUMIC_CONTRACTION  such  that 

[  isovol _ contract. time_int. end _ time  = 

max_ejection.time_int.start_time  , 

isovol _ contract. subj  =  self.subj  ]; 

max_ejection  :  N_MAXIMUM_EJECTION  such  that 
[  max_ejection.subj  =  self.subj  , 

max_ejection.time_int.  end _ time 

=  red_ejection.time_int.start_time  ]; 

red _ ejection  :  N_REDUCED_EJECTION  such  that 

[  red_ejection.subj  =  self.subj  ]; 
dependents 

time_int  :  with 

start_time  <-  isovol _ contract. time_int.start_time  , 

end _ time  *-  red _ ejection. time_int. end _ time  , 

duration  default  0.27  *  AD  /  0.8  such  that 
[  duration  ^  0.38  *  AD  /  0.8  , 
duration  ^  0.17  *  AD  /  0.8  ]; 
similarity  links  : 

sim _ linkl  :  ISCH_CAP_SYST0LE 

for  differences  : 

dl  :  T00_SL0W  where 

[  subj  instance_of  N_AP_SEG  , 
time_int  =  max_ejection.time_int  ], 
d2  :  T00_SL0W  where 

[  subj  instance_of  N_AP_SEG  , 
time_int  =  red_ejection.time_int  ]; 
bound  by  : 

subj  ->  subj  , 

isovol _ contract  -+  isovol _ contract  , 

max_ejection.anterior_mot  ->  max _ ejection. anterior _ mot  , 

max_ejection.posterior_mot  -*  max_ejection.posterior_mot  , 
red_ejection.antcrior_mot  -»  red_ejection.anterior_mot  , 
red_ejection.posterior_mot  ->  red_ejection.posterior_mot  ;; 

end 
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frame  N _ DIASTOLE  is„ a  SEQUENCE  with 

prerequisites 

subj  :  N_LV  ; 

isovo] _ relax  :  N„ISQV0LUMIC__RELAXAT10N  such  that 

[  isovoL_relax.subj  =  self.subj  ]; 

rapid _ fill  :  N_RAPID__F1LL  such  that 

[  rapid_fill.subj  =  self.subj  , 
rapid_fiil.time_int.start_time  = 

isovo) _ relax. time_int. end _ time  , 

rapid _ fill.  time_int.  end _ time 

=  diastasis. time_int.start_time  ]; 
diastasis  :  N_DIASTASIS  such  that 

[  diastasis. time_int. end _ time 

=  fill _ atrial _ cont.time_int. start_time  , 

diastasis. subj  =  self.subj  ]; 

fill _ atrial _ cont  :  N_FILL_ATRIAL_  CO  NT  such  that 

[  fill _ atrial_cont.subj  =  self.subj  ]; 

dependents 

time_int  :  with 

start_time  «-  isovo! _ relax. time_int.start_time  , 

end _ time  <-  fill _ atrial _ cont. time_int. end _ time  , 

duration  default  0.53  *  AD  /  0.8  such  that 
duration  ^  0.60  *  AD  /  0.8  , 
duration  ^  0.36  *  AD  /  0.8  ]; 
similarity  links  : 

sim _ linkl  :  1SCH_CAP__DIAST0LE 

for  differences  : 

dl  :  N  0_N_1  S  0  V0  LU  MI  C_RE  LAX  ATI  0  N  where 

[  prereq_part  instance_of  ISCR_CAP_IS0V0L_RELAX  ], 
d2  :  T00_SL0W  where 

[  subj  instance_of  N_AP_SEG  , 

time_int  =  rapid _ fill.time_int  ], 

d3  :  T00_SL0W  where 

[  subj  instance_of  N_AP_SEG  , 
time_int  =  diastasis. time_int  ], 
d4  :  T00_SL0W  where 

[  subj  instance_of  N_AP_SEG  , 

time_int  =  fill _ atrial _ cont.time_int  ]; 

bound  by  : 

subj  ->  subj  , 

rapid _ fill.anterior_mot  -»  rapid _ fill.anterior_mot  , 

rapid_fill.posterior_mot  ->  rapid__fill.posterior_m.ot  , 
diastasis. anterior_mot  -»  diastasis. anterior_mot  , 
diastasis. posterior_mot  ->  diastasis. posterior_mot  , 

fill_atrial _ cont.  anterior_mot  -»  fill _ atrial _ cont.anterior_mot  , 

fill_atrial_cont.posterior_mot  ->  fill_atrial_cont.posterior_mot 

end 
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Systole  and  diastole  are  in  turn  decomposed  into  their  phases  so  that  the 
resultant  7  phases  in  total  reflect  those  found  in  the  cardiology  literature  [Silber75]. 
These  frames  state  that  in  each  phase,  all  segments  move  in  the  same  direction  (to¬ 
wards  or  away  from  the  centroid)  and  with  the  same  relative  speed.  Note  that  speeds 
between  phases  are  not  compared. 

frame  N_IS0V0LUMIC_C0NTRACT10N  is_ a  ARE A_ CHANGE  -with 
prerequisites 

subj  :  N_LV  ; 

end _ a  :  such  that 

[  end _ a  ^  start_a  *  0.9  , 

end_a  S  start_a  *  1.1  ]; 
dependents 

time_int  :  urith 

duration  default  0.5  *  AD  /  0.3  such  that 

duration  S  0.07  *  AD  /  0.8  , 
duration  ^  0.03  *  AD  /  0.8  ]; 

end 
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frame  N„ISOVOLUMIC„RELAXATION  a  SIM  UE_M  OT_P  ARTS  with 

prerequisites 

subj  :  N._LV  ; 

anterior_mot  :  N_ANT_ISO_RELAX  such  that 
[  anterior_mot.subj  part_of  self.subj  ]; 

apical _ mot  :  N_AP_ISO_RELAX  such  that 

[  apical_mot.subj  part„of  self.subj  ]; 
posterior_mot  :  N_POST_ISO_RELAX  such  that 
[  posterior_mot.subj  part_of  self.subj  ]; 
dependents 

time_int  :  with 

start_time  *-  anterior_mot.time_int.start_time  such  that 
[  same(anterior_mot. time_int.start_time  , 

apica1 _ mot.time_int.start_time  , 

posterior_mot.time_int.start_time  )  ], 

end _ time  «-  anterior_mot. time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  . 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ], 

duration  default  0.12  *  AD  /  0.8  such  that 
[  duration  ^  0.13  *  AD  /  0.8  , 
duration  ^  0.10  *  AD  /  0.8  ]; 
similarity  links 

sim _ link  1  :  1SCH_CAP_IS0V0L_RELAX 

for  differences  : 

dl  :  TO 0_MU CH_M OTI 0 N  where 

[  subj  instance_of  N _ AP_SEG  , 

time_int  =  anterior_mot.time_int  ]; 
bound  by  : 

anterior_mot  -»  anterior_mot  , 
posterior_mot  ->  posterior_mot  ;; 

end 
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frame  N_MAXIMUM_EJECT10N  is_a  CONTRACT,  SIMUL_MOT_PARTS  with 
prerequisites 

subj  :  N_LV  ; 

anterior_mot  :  N_ANT_MAX_EJ  such  that 
[  anterior_mot.subj  part_of  self.subj  , 
for  dirl  :  TOWARDS  -where 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self. subj. centroid  ]]; 

apical _ mot  :  N _ AP _ MAX_EJ  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

(apical _ mot. translate. speed  <  anterior_mot. translate. speed  or 

apical _ mot. translate. speed  <  posterior_mot. translate. speed  )  and 

posterior_mot.  translate. speed  =  anterior_mot. translate,  speed 
exception  TOO_SLOW  with 

[  subj  <-  apical _ seg  , 

time_int  <-  self. time_int  ], 
for  dir2  :  TOWARDS  where 

[  dir2.subj  =  apical _ mot. subj  , 

dir2.ref  =  self.subj. centroid  ] 
exception  N0T_T0W4RDS  with 

[  subj  «-  apical _ seg  , 

time_int  <-  self. time_int  ]]; 
posterior_mot  :  N_POST_MAX_EJ  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir3  :  TOWARDS  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self.subj. centroid  ]]; 

dependents 

time_int  :  with 

start_time  <-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 
apical_mot.time_int.start_time  , 
posterior_mot.time_int.start_time  )  ], 

end _ time  <-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  . 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ], 

duration  default  0.09  *  AD  /  0.8  such  that 
[  duration  ^  0. 13  *  AD  /  0.8  , 
duration  <=?  0.056  *  AD  /  0.8  ]; 

end 


Note  that  in  the  above  frame  the  exception  NOT_TOWARDS  could  be  used  in  a 
similarity  link  to  trigger  the  DYSKINETIC_LV_CYCLE  frame.  Dyskinesis  is  term  used 
for  abnormalities  of  motion  due  primarily  to  wrong  trajectories  of  segmental  motion. 
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frame  N_REDUCED_EJECTION  is_a  CONTRACT,  SIMUL_MOT_PARTS  with 
prerequisites 

subj  :  N_LV  ; 

anterior_mot  :  N_ANT_RED_EJ  such  that 
[  e.nterior_mot.subj  part_of  self.subj  , 
for  dirl  :  TOWARDS  "where 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self. subj. centroid  ]]; 

apical _ mot  :  N _ A.P_RED_EJ  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

(apical _ mot. translate. speed  <  anterior_mot. translate. speed  or 

apical _ mot. translate. speed  <  posterior_mot. translate. speed  )  and 

posterior_mot. translate. speed  =  anterior_mot.  translate,  speed 
exception  TOO_SLOW  with 

[  subj  «-  apical _ seg  , 

time_int  self.  time_int  ], 
for  dir2  :  TOWARDS  where 

[  dir2.subj  =  apical _ mot. subj  , 

dir2.ref  =  self.subj. centroid  ] 
exception  N OT_TO W ARE) S  with 
[  subj  *-  apical_seg  , 
time_int  «-  self.time_int  ]]; 
posterior_mot  :  N_POST_RED_EJ  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir3  :  TOWARDS  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self. subj. centroid  ]]; 


dependents 

time_int  :  with 

start_time  <-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 

apical _ mot.time_int.start_time  , 

posterior_mot.time_int.start_time  )  ], 

end _ time  *-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_rnot.time_int.end _ time  , 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ], 

duration  default  0.13  *  AD  /  0.8  such  that 
[  duration  ^  0.18  *  AD  /  0.8  , 
duration  ^  0.08  *  AD  /  0.8  ]; 


end 
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frame  N_RAPID_FILL  is_a  EXPAND,  SIMUL_MOT_PARTS  with 
prerequisites 

subj  :  N_LV  ; 

anterior_mot  :  N_ANT_RAP_FILL  such  that 
[  anterior_mot.subj  part_of  self.subj  , 
for  dirl  :  AWAY_FROM  where 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self.subj. centroid  ]]; 

apical _ mot  :  N_AP_RAP_FILL  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

(apical _ mot. translate. speed  <  anterior_mot. translate. speed  or 

apical _ mot. translate. speed  <  posterior_mot. translate. speed  )  and 

posterior_mot. translate. speed  =  anterior_mot.  translate,  speed 
exception  TOO_SLOW  with 

[  subj  <-  apical _ seg  , 

time_int  «-  self.  time_int  ], 
for  dir2  :  AWAY_FROM  where 

[  dir2.subj  =  apical _ mot. subj  , 

dir2.ref  =  self. subj. centroid  ] 

exception  NOT _ AWAY _ FROM  with 

[  subj  «-  apical _ seg  , 

time_int  <-  self.time_int  ]]; 

posterior_mot  :  N _ POST_RAP_FILL  such  that 

[  posterior_mot.subj  part_of  self.subj  , 
for  dir 3  :  AWAY_FROM  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self. subj. centroid  ]]; 

dependents 

time_int  :  with 

start_time  «-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 
apical_mot.time_int.start_time  , 
posterior_mot.time_int.start_time  )  ], 

end_time  «-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ], 

duration  default  0.11  *  AD  /  0.9  such  that 
[  duration  ^  0. 15  *  AD  /  0.8  , 
duration  ^  0.07  *  AD  /  O.B  ]; 

end 
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frame  N_DIASTASIS  is_a  EXPAND,  SIMTJL_MOT_PARTS  with 
prereq-u.isi.tes 

subj  :  N_LV  ; 

anterior_mot  :  N_ANT_DIASTAS1S  such  that 
[  anterior_mot.subj  part_of  self.subj  , 
for  dirl  :  AWAY_FROM  -where 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self.subj. centroid  ]]; 

apical _ mot  :  N _ AP_DLASTAS1S  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

(apical _ mot. translate. speed  <  anterior _ mot. translate. speed  or 

apical _ mot. translate. speed  <  posterior__mot. translate. speed  )  and 

posterior_mot.  translate,  speed  =  anterior_mot. translate. speed 
exception  TOO_SLOW  -with 

[  subj  <-  apical _ seg  , 

time_int  <-  self.  time_int  ], 
for  dir 2  :  AWAY_FROM  where 

[  dir2.subj  =  apical _ mot. subj  , 

dir2.ref  =  self. subj. centroid  ] 
exception  NOT_AWAY_FROM  with 

[  subj  <-  apical _ seg  , 

time_int  «-  self. time_int  ]]; 
posterior_mot  :  N_POST_DIASTASIS  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir3  :  AWAY_FROM  where 

[  dir3.subj  =  posterior__mot.subj  , 
dir3.ref  =  self.subj. centroid  ]]; 

dependents 

time_int  :  -with 

start_time  «-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 

apical _ mot.time_int.start_time  , 

posterior_mot.time_int.start_time  )  ]  , 

end_time  <-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical_mot.time_int.end_time  , 

posterior_mot.time_int.end _ time  )  ], 

duration  default  0.19  *  AD  /  0.8  such  that 
[  duration  ^  0.27  *  AD  /  0.8  , 
duration  ^  0.12  *  AD  /  0.8  ]; 

end 
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frame  N_FILL_ATRLAL_CONT  is_a  EXPAND,  SIMUL_MOT_PARTS  with, 
prerequisites 

subj  :  N_LV  ; 

anterior_mot  :  N_ANT_FILL_ATR_CONT  such  that 
[  anterior_mot.subj  part_of  self.subj  , 
for  dirl  :  AWAY_FROM  where 

[  dirl. subj  =  anterior _ mot. subj  , 

dirl. ref  =  self.subj. centroid  ]]; 

apical _ mot  :  N_AP_FILL _ ATR_CONT  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

(apical _ mot. translate. speed  <  anterior_mot. translate. speed  or 

apical _ mot. translate. speed  <  posterior_mot. translate. speed  )  and 

posterior_mot. translate. speed  =  anterior_mot. translate,  speed 
exception  TOO_SLOW  with 

[  subj  *-  apical _ seg  , 

time_int  «-  self.time_int  ], 
for  dir 2  :  AWAY_FROM  where 

[  dir2.subj  =  apical _ mot.subj  , 

dir2.ref  =  self. subj. centroid  ] 

exception  NOT _ AWAY_FROM  with 

[  subj  *-  apical _ seg  , 

time_int  <-  self.time_int  ]]; 
posterior_mot  :  N_POST_FILL_ATR_CONT  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir3  :  AWAY_FROM  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self. subj. centroid  ]]; 


dependents 

time_int  :  with 

start_time  «-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 
apical_mot.time_int.start_time  , 
posterior_mot.time_int.start_time  )  ], 

end _ time  «-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ], 

duration  default  0.11  *  AD  /  O.B  such  that 
[  duration  S  0.15  *  AD  /  0.8  , 
duration  ^  0.068  *  AD  /  0.8  ]; 

end 
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The  motions  of  each  individual  segment  for  each  phase  form  the  next  level  of 
the  LV  cycle's  PART_OF  hierarchy.  Various  combinations  of  translations,  rotations  and 
length  changes  would  constitute  these  motions  and  therefore  they  would  fall  under  the 
simultaneous  motions  node  in  the  IS _ A  hierarchy.  These  motions  are  not  known  pre¬ 

cisely  and  for  now  these  frames  are  null.  However,  the  motions  of  each  segment  during 
isovolumic  relaxation  are  known  [Gibson76].  Both  apical  and  posterior  segments  move 
in  the  same  ways.  They  may  move  inwards  up  to  2mm  or  outwards  up  to  6mm.  The 
anterior  segment  must  be  motionless.  The  frames  defining  these  motions  are: 

frame  N_AP__ISO_RELAX  is_ a  SIMUL_MOT  with 
prerequisites 

subj  :  N _ AP_SEG  ; 

mot  :  TRANSLATE  such  that 

[  (  for  dirl  :  TOWARDS  where 
[  dirl. subj  =  self.subj  , 
dirl. ref  =  super_part(self). centroid  , 
speed  <  2  /  time_int. duration  ] 
or 

for  dir 2  :  AWAY_FROM  where 
[  ,  dir2.subj  =  self.subj  , 
dir2.ref  =  super_part(self). centroid  , 
speed  <  6  /  time__int. duration  ]  ) 
exception  TOO_MUCH_MOTION  urith 
[  subj  «-  self.subj  , 
time_int  *-  self. time_int  ]]; 

end 


”super_part"  is  a  function  and  returns  the  concept  of  which  its  argument  is  a 
PART_OF. 

frame  N_POST_ISO_RELAX  is_a  SIMUL_MOT  with 
prerequisites 

subj  :  N_POST_SEG  ; 

mot  :  TRANSLATE  such  that 

[  (  for  dirl  :  TOWARDS  where 
[  dirl. subj  =  self.subj  , 
dirl. ref  =  super_part(self). centroid  , 
speed  <  2  /  time__int. duration  ] 
or 

for  dir2  :  AWAY_FROM  where 
[  dir2.subj  =  self.subj  , 
dir2.ref  =  super_part(self). centroid  , 
self.speed  <  6  /  self. time_int. duration  ]  ) 
exception  TOO_MUCH_MOTION  with 
[  subj  *-  self.subj  , 
time_int  «-  self.time_int  ]]; 

end 
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frame  N_ANT_IS 0_RELAX  is_o  NON_MOTION 
end 

The  definition  of  the  motions  during  the  isovolumic  relaxation  phase  are  very 
important  from  a  physiological  point  of  view.  This,  and  the  isovolumic  contraction 
phase  are  the  only  portions  of  the  cycle  during  which  the  LV  is  isolated  from  the  peri¬ 
phery  (the  remainder  of  the  circulatory  system),  and  thus  can,  in  a  sense,  be  studied 
independently.  In  addition,  the  isovolumic  relaxation  phase  is  not  a  passive  one  and  re¬ 
quires  energy.  The  way  in  which  the  LV  responds  during  this  phase  is  a  good  indicator 
of  muscle  function. 

The  remainder  of  the  frames  defining  segmental  motions  during  the  other 
phases  are  set  to  null  for  the  moment  until  further  research  can  supply  us  with  more 
quantitative  data. 

Several  similarity  links  to  the  ischemic  central  apex  cycle  frames  were  given 
above.  An  example  of  another  disease  is  given  below  in  order  to  further  illustrate  the 
links.  This  link,  for  which  the  data  was  found  in  a  cardiology  journal  [Sanderson77], 
and  used  directly,  is  that  which  relates  the  normal  left  ventricular  cycle  to  that  of  a  hy¬ 
pertrophic  ventricle  (hypertrophic  cardiomyopathy).  Two  separate  links  would  appear 
in  the  knowledge  base  from  the  normal  case  to  the  hypertrophic  case  because  of  the 
exception  handling  mechanism’s  properties.  As  stated  in  the  representation  chapter, 
exceptions  are  handled  by  the  frame  which  is  the  parent  of  the  one  in  which  the  excep¬ 
tion  occurred.  Due  to  the  particular  sequence  of  anomalies  of  hypertrophic  ventricles, 
and  to  the  particular  definitions  of  the  phases,  two  links  are  necessary:  one  in  the 
N_LV_CY CLE  frame  and  one  in  the  N_DIASTOLE  frame.  They  appear  below  in  that  ord¬ 
er. 

similarity  links 

sim _ linkl  :  HYPERTROPHIC_CARDIOMYOPATHY 

with  similarities  : 

subj.lat_axis.arclen  =  hc.subj.lat_axis.arclen 
at  isovol _ contract.time_int.start_time  ; 

for  differences  : 

x  :  T00_SH0RT  -where  [  subj  =  self.lat_axis  , 

time_int  =  isovol_relax.time_int  ]  , 
y  :  TOO_LONG_DUR  where  [  subj  =  self  , 

time_int  =  isovol _ relax. time_int  ]  ; 
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bound _ by  : 

isovol _ contract.  -*  isovol_contract  , 

systole  ->  systole  , 
subj  -»  subj  ; 

sim _ link4  :  HYPERTROPHIC_CARDIOMYPATHY 

for  differences  : 

z  :  TQO__LONG_DUR  -where  [  subj  =  self  , 

time_int  =  rapid _ fill.time_int  ]; 

bound _ by  : 

isovol _ contract  -*  isovol _ contract  , 

subj  -*  subj  , 

isovol _ relax  -»  isovol _ relax  , 

systole  -»  systole  ; 

The  values  for  determination  of  when  these  exceptions  occur  are  also  given  in 
that  paper  and  would  be  included  in  the  appropriate  slot  descriptions,  "he"  is  the 
name  of  the  frame  concept  for  the  hypertrophic  cardiomyopathy.  The  TOO_SHORT  ex¬ 
ception  (used  only  for  comparisons  of  physical  dimension)  is  raised  for  LV  width  (length 
of  the  lateral  axis)  if  its  dimension  falls  between  2.0  and  2.8  cm.  The  normal  values  are 
3.3  and  5.1  cm.  at  the  start  of  the  isovolumic  relaxation  phase.  The  T00_L0NG_DUR 
exception  (that  is,  duration  is  too  long  for  an  event)  is  raised,  in  the  first  case  if  the  iso¬ 
volumic  relaxation  phase’s  duration  is  between  100  and  180  msec.  The  normal  values 
are  79  to  107  msec.  In  the  second  case,  it  is  raised  if  the  duration  is  between  96  and 
224  msec,  while  the  normal  values  are  110  to  150  msec.  Note  that  in  the  first 
T00_L0NG_DUR  exception  ("y"),  the  values  overlap  with  the  normal  case.  This  means 
that  both  frames  would  be  active  and  subsequent  data  would  be  used  to  choose  the 
better  of  the  two. 

There  are  several  additional  domain  related  considerations  which  must  play  a 
part  in  the  definition  of  the  knowledge  base. 

a)  LV  orientation  and  size  -  Orientation  is  not  important  because  all  descrip¬ 
tions  are  relative  to  the  LV  and  its  shape  feature  axes.  Size,  however,  is  important  be¬ 
cause  different  size  ventricles  exhibit  different  motion  characteristics.  In  addition, 
thick  walled  and  thin  walled  ventricles  exhibit  very  different  motions.  Thus,  these 
different  types  require  different  motion  frames.  The  LV  size  and  wall  thickness  will  be 
input  to  the  system  before  analysis.  This  information  can  be  used  so  that  only  the 
relevant  portions  of  the  KB  are  initially  considered. 
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b)  Rocking  Motion  -  In  addition  to  the  contraction-relaxation  motion  patterns 
of  the  left  ventricular  wall,  the  heart  may  also  "rock"  or  "sway"  within  the  chest  cavity. 
A  frame  to  define  rocking  motion  may  be  set  up  using  the  following  characterization. 
The  motion  involves  rotation  of  the  LV’s  length  axis  about  some  point  in  the  sinus  of  val¬ 
salva.  The  rotation  must  be  cyclic  and  such  a  motion  is  the  definition  of  the  verb 
"SWAY".  However,  the  fact  that  it  is  superimposed  onto  the  LV  contraction  and  expan¬ 
sion  pattern  must  be  taken  care  of.  This  may  be  done  by  including  rocking  in  the 
LV_cycle  frame  and  distinguishing  between  the  two  frames  by  naming  them 
"LV__CYCLE"  and  "LV_CYCLE_WITH_ROCKING".  The  rocking  seems  to  have  the  same 
cycle  time  as  the  beat.  Thus,  if  the  system  has  hypothesized  a  rocking  motion  of  the 
LV,  it  can  be  added  to  the  regular  motion  before  comparing  it  to  the  description  of  the 
observed  LV.  Then  it  may  be  subtracted  out.  In  general,  if  a  simple  motion  is  superim¬ 
posed  onto  an  aggregate  motion,  the  motion  frame  will  contain  the  events  for  the 
object’s  parts  and  events  for  the  superimposed  motions.  This  may  be  a  superimposed 
translation  or  rotation,  or  a  change  of  physical  properties  such  as  a  change  of  area. 
The  information  in  this  last  case  would  be  redundant,  because  it  is  the  result  of  the 
sum  of  the  motions  of  the  object’s  parts. 

c)  Abnormality  Regions  -  The  characteristics  of  regions  of  the  LV  wall  which 
exhibit  abnormal  motion  patterns,  regardless  of  the  type  of  abnormality  (akinesis, 
dyskinesis,  hyperkinesis,  etc.),  are  also  stored  in  the  knowledge  base.  Abnormality  re¬ 
gions  do  not  always  fall  wholly  within  one  segment  and  if  they  span  two  or  more  seg¬ 
ments,  their  characteristics  may  be  different.  However,  these  characteristics  may  be 
defined  for  each  abnormality  relative  to  its  position  on  the  LV  wall.  In  addition,  an  ab¬ 
normality  spanning  two  regions  has  different  properties  again.  The  boundary  condi¬ 
tions  between  abnormal  and  normal  tissue  may  differ  depending  on  size  and  location  of 
abnormality.  Therefore,  it  has  been  found  necessary  to  have  separate  frames  for  each 
of  these  abnormality  types  to  ensure  that  the  above  considerations  are  taken  into  ac¬ 
count. 
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Abnormal  motion  patterns  will  be  restricted  to  the  motion  patterns  of  three 
points  on  each  segment:  the  end  points  and  the  mid-point.  Therefore,  frames  are 
defined  for  each  of  these  different  regions  and  for  combinations  of  them.  The  regions 
will  be  known  as  the  anterior,  central  and  posterior  regions  of  each  segment  (prefixes 
used  will  be  A,  C.  and  P). 

One  large  class  of  abnormalities  has  been  defined  using  data  found  in  [Gib- 
son76].  This  class  is  that  for  the  motions  of  the  LV  segments  experiencing  ischemia. 
Ischemia  occurs  in  varying  degrees  depending  on  the  severity  and  duration  of  disease. 
The  main  cause  is  occlusion  or  narrowing  of  the  coronary  arteries  -  the  arteries  which 
provide  nutrients  for  the  LV  muscle.  Clearly,  if  the  artery  is  blocked  to  some  degree, 
the  LV  muscle  supplied  by  it  cannot  be  perfused  properly  and  therefore  its  action  is 
impaired.  If  such  a  problem  exists  for  long  enough  a  period,  the  muscle  becomes  dam¬ 
aged  irreversibly.  In  most  cases  however,  the  damage  is  reversible  if  the  cause  is 
remedied  (coronary  bypass  surgery). 

Ischemia  would  be  recognizable  in  cineangiograms  as  LV  wall  segments  which 
have  a  reduced  amplitude  of  motion  relative  to  the  other  normal  segments.  Also, 
research  has  revealed  that  during  the  isovolumic  relaxation  phase,  small  changes  in 
motion  from  the  normal  case  are  positive  indicators  of  ischemic  disease  in  that  seg¬ 
ment.  The  duration  of  the  isovolumic  relaxation  phase  is  also  lengthened.  In  addition, 
in  ischemic  heart  disease,  the  amplitude  of  motion  during  both  systole  and  diastole  is 
reduced  from  the  normal  motion  for  the  affected  segment. 

The  PART_OF  hierarchy  for  ischemic  LV  cycles,  only  for  the  central  apex 
case,  is  shown  in  Fig.  5.7.  The  1S_A  relationships  are  noted  in  the  frame  definitions  and 
are  summarized  in  Fig.  5.8.  The  frames  for  ischemic  cycles  are  given  in  Appendix  B. 
Only  those  for  central  apex  ischemia  are  given  -  the  ones  for  the  other  segments  are 
very  much  the  same,  and  it  should  be  clear  how  to  combine  parts  in  order  to  define 
frames  for  combinations  of  segments.  In  addition,  the  frames  for  ischemia  are  related 
back  to  the  normal  case  using  similarity  links. 
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ISCHEMIC  CAP  LV  CYCLE 
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FIGURE  5 „ 7  ISCHEMIC  APEX  LV  CYCLE  PART  OF  HIERARCHY 


FIGURE  5.8 


ISCHEMIC  APEX  LV  CYCLE  MOTION  IS  A  HIERARCHY 


CHAPTER  6 


A  KNOWLEDGE-BASED  FEEDBACK  MODEL  FOR  ANALYSIS  OF 


TIM  E-VAR  YING  IMA  GER  Y 


6. 1  Overview 

The  recognition  paradigm  that  we  will  use  was  set  out  in  Chapter  2  and  it  will 
be  elaborated  now.  The  recognition  mechanism  relies  heavily  on  relaxation  labelling 
methods,  both  at  the  image  level  and  at  the  conceptual  level.  Control  is  structured  as 
in  feedback  control  systems;  a  diagram  of  the  proposed  structure  is  shown  in  Fig.  6.1. 

An  image  analysis  component  using  high  level  expectations  about  object  lo¬ 
cations  in  the  image  sequence  produces  the  essential  kineses  for  each  object  between 
images  and  corresponds  to  the  "feedback  transducer"  in  control  theory.  The  matcher 
compares  the  essential  kineses  for  each  object  with  the  motion  classes  hypothesized 
for  that  object.  Any  discrepancies  are  recorded  as  exception  frames,  which  are  passed 
on  to  the  next  component  in  the  cycle.  Information  about  general  motion  classes  is  ob¬ 
tained  from  the  generic  knowledge  base  ,  which  is  structured  in  the  form  outlined  in 
Chapter  3,  while  specific  hypotheses  for  the  objects  in  the  actual  film  sequence  are 
found  in  the  specific  knowledge  base.  The  specifics,  or  instances  are  related  to  the 
generic  information  through  the  instance_of  relationship.  Exceptions  generated  by 
the  matcher  are  used  to  activate  similarity  links  by  the  change  of  attention  mechan¬ 
ism.  Active  similarity  links  point  to  hypotheses  which  should  be  taken  into  considera¬ 
tion  for  some  object  because  the  current  hypothesis  does  not  adequately  describe  the 
motion  the  object  exhibits.  The  focus  of  attention  mechanism  then  rates  the  hy- 
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Figure  6.1  The  Knowl edge  -  Based  Feedback  Recognition  Model 
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potheses  using  certainty  factors  as  a  measure  of  the  matching  progress  and  con¬ 
sistency  of  each  hypothesis.  The  best  hypotheses  are  then  used  in  expectation  gen¬ 
eration  for  the  probable  locations  of  each  object  in  the  next  image. 

The  remainder  of  this  chapter  discusses  each  of  these  components  in  the  ord¬ 
er  in  which  they  were  presented  above. 

6.2  Image  Analysis  -  Extracting  the  Data 

The  image  analysis  component  consists  of  two  processes  that  are  executed 
sequentially:  (l)  a  process  that  finds  objects  in  each  image  based  on  predictions  from 
the  hypothesized  motions  of  each  object,  and,  (2)  a  process  that  computes  the  inter- 
image  descriptions  of  the  motions  detected  for  each  object  between  consecutive  im¬ 
ages.  The  first  is  based  upon  the  relaxation  method  of  [Zucker77],  modified  to  ac¬ 
count  for  the  expectations.  The  second  is  based  upon  a  set  of  heuristics  designed  for 
our  particular  domain.  However,  relaxation  labelling  could  also  be  used  here  for  the 
shape  matching  as  in  [Davis77].  This  is  left  for  future  work. 

6.2.  1  Computing  the  Essential  Trace 

The  first  step  in  image  analysis  is  the  determination  and  description  of  object 
boundaries.  For  each  image,  alter  the  first  one,  a  predicted  search  window  is  used  to 
aid  in  determination  of  this  boundary.  Additional  descriptors  are  then  added  for  each 
object.  This  information  is  termed  the  essential  trace  of  the  object. 

The  computation  of  the  essential  trace  is  done  in  two  steps.  The  first  is  the 
determination  of  the  edges  of  the  object  in  the  image,  and  may  be  termed  the  compu¬ 
tation  of  the  ram  essential  trace.  Because  of  the  poor  quality  of  the  images  of  our 
domain  ,  such  edges  are  often  difficult  (or  impossible)  to  find.  Thus  the  system's  ex¬ 
pectations  are  used  to  limit  the  area  over  which  edges  are  searched  for,  and  also  to 
bias  the  system  towards  edge  segments  with  certain  orientations.  Objects  which  the 
system  considers  as  "fixed"  (those  that  cannot  move  under  the  assumption  of  no  ob¬ 
server  motion),  have  no  expectations  associated  with  them  from  image  to  image. 
Those  objects  which  can  move  ("mobile”),  even  if  no  hypothesized  motions  have  been 
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associated  with  them,  always  have  expectations  for  each  image.  When  the  expectations 
do  not  agree  with  reality,  better  estimates  for  the  edges  are  found  by  relaxing  the  con¬ 
straints  on  the  expectations. 

The  second  step  is  an  analysis  of  the  raw  trace  in  which  descriptors  are  at¬ 
tached  to  the  elements  of  the  trace.  These  descriptors  are  shape  features,  and  size 
and  location  information.  The  final  essential  trace  thus  consists  of  the  raw  edge  ele¬ 
ments  along  with  their  associated  descriptors.  The  style  of  the  descriptors  is  in  the 
same  vein  as  the  kinds  of  descriptors  found  in  [Marr76]. 

Clearly  the  first  image  is  a  special  case,  because  no  high  level  expectations 
are  possible.  The  model  assumes  that  a  conceptual  description  is  given  for  this  first 
image,  which  may  be  provided  either  by  the  user  or  computed  by  the  methods  avail¬ 
able  for  single  image  analysis.  Its  form  will  be  the  same  as  that  determined  for  each 
object  in  the  successive  images. 

6. 2. 1.1  The  Ra-w  Trace 

The  method  for  determining  the  raw  trace  follows  the  relaxation  labelling  pro¬ 
cess  outlined  in  [Zucker77].  Modifications  to  the  process  are  necessary,  however,  be¬ 
cause  predictions  for  edge  location  are  provided  by  the  higher  levels  of  the  system. 
These  predictions  enable  the  process  to  be  much  more  efficient  than  without  guidance, 
and  also  enable  it  to  discover  edges  that  would  otherwise  be  very  difficult  to  find. 

The  input  to  the  first  phase  of  the  process  is  simply  the  list  of  connected  edge 
segments  that  form  the  predicted  outline  of  the  object  of  interest.  Associated  with 
each  segment  is  a  region  radius.  This  radius  defines  the  extent  of  the  search  space  in 
the  image  for  that  segment.  In  other  words,  it  defines  the  sub-image  over  which  relax¬ 
ation  is  to  be  performed.  In  addition,  a  starting  point  on  the  outline  and  a  possibly 
varying  search  length  are  provided.  For  closed  loop  outlines,  this  starting  point  is  not 
crucial  as  long  as  it  is  consistently  determined.  For  left  ventricles,  which  have  open 
loop  outlines,  the  left  upper  point  of  the  aorta  is  used  as  the  starting  point.  The  length 
of  outline  to  be  searched  for  at, any  one  instant  depends  on  the  confidence  of  the  pred- 
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iction:  the  smaller  the  confidence  of  the  actual  edge’s  location,  the  shorter  the  search 
length. 

Beginning  with  the  starting  point,  the  specified  length  of  predicted  outline  is 
taken  and  a  region  of  specified  radius  is  created  for  it.  Each  point  in  the  region  is  la¬ 
belled  with  the  edge  orientation  of  the  portion  of  the  outline  closest  to  it.  Next,  for 
each  point  in  the  region,  a  5  X  5  gradient  edge  operator  is  applied  in  each  of  B  possible 
edge  orientations,  as  well  as  a  "no-edge"  operator.  The  no-edge  operator  is  basically  a 
detector  of  variance  among  the  responses  for  the  eight  edge  orientations.  Its  response 
is  high  if  there  is  little  difference  among  these  responses  and  low  if  there  is  a  peak  in 
the  response  function.  The  responses  of  the  operators  are  associated  with  the  point  as 
certainties  and  are  supplied  to  a  prediction  guided  relaxation  labelling  process  for 
determination  of  the  object’s  edge. 

Explicit  edge  hypotheses  are  represented  as  labels  attached  to  image  points. 
The  labels  are  ordered  by  certainty  factors  that  specify  for  each  label  how  likely  it  is  to 
be  the  correct  one  for  that  point.  It  is  these  factors  that  are  updated  by  the  relaxation 
labelling  process.  The  updating  rule  that  is  used  to  update  the  certainty  factors  (from 
[Zucker77])  is: 

.  u  *)*[l  +  g(*.  x)] 

*  norm 

where  p’(i,\)  is  the  new  certainty  value  of  point  i  having  labe  IX.  q(i,  \)  is  the  contri¬ 
bution  from  the  neighbouring  hypotheses  (there  are  N  of  them)  and  is  given  by: 

N  9 

7  (*.  X)  =  S  w  (».*)*  E  comp  (X,  \j)*p  (k.  \j) 

t  =  i  i  =  i 

norm  is  the  normalizing  factor  and  is  given  by: 

9 

norm  =^P(^  *)*  [1+7  (i.  X)] 
x=i 

and  w  is  the  weighting  function  of  the  contribution  from  one  point  to  its  spatial  neigh¬ 
bour,  that  is,  the  amount  of  contribution  from  point  k  to  point  i.  The  sum  of  the  con¬ 
tributions  must  be  unity.  Finally,  comp  is  the  compatibility  function  between  two  hy¬ 
potheses  or  labels. 
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Predictions  enter  the  relaxation  process  in  two  ways:  the  search  region 
(described  in  the  previous  paragraphs)  and  the  expected  edge  orientation.  Relaxation 
is  only  applied  to  the  points  for  which  the  gradient  operator  has  produced  responses, 
i.e.,  in  the  search  region.  The  expected  orientation  is  included  in  the  computation  by 
partitioning  search  regions  into  mosaics  of  sub-regions,  and  then  associating  different 
orientation  biases  with  each  of  these  sub-regions.  These  biases  cause  the  relaxation 
updating  function  to  favour  edge  labels  of  the  expected  orientation  (and  those  immedi¬ 
ately  compatible  with  that  orientation).  Such  information  is  coded  into  a  "strength” 
function.  Suppose  the  expected  edge  orientation  were  "4".  The  corresponding  strength 
function  for  all  the  possible  edge  labels  (excluding  the  no-edge  label)  would  be: 

A  01234567 
s(4,A)  0.2  0.4  0.8  1.0  1.0  1.0  0.8  0.4 

where  A  indicates  the  label  and  s(4,A)  indicates  the  strength  function  for  predicted 
orientation  4.  In  general,  the  strength  function  has  a  range  of  0.0  to  1.0. 

Thus,  edges  that  are  oriented  in  approximately  the  same  direction  as  the  ex¬ 
pectation  are  favoured,  while  orthogonal  ones  are  biased  against.  The  relaxation  up¬ 
dating  function  in  [Zucker77]  is  re-defined  to  include  the  above  orientation  biases  as 
follows: 

.  (ii  \)*[l  +  g(i  A)1*S  (d.  A) 

norm 

norm  =  A)*  [l  +  g  (i,  A)]*s  (d,  A) 

A=1 

where  s(d, \)  is  the  strength  function.  The  parameters  of  s  are  the  predicted  orienta¬ 
tion  d  and  the  label  under  consideration  A. 

It  is  possible  for  obstructions  such  as  ribs  or  the  diaphragm  to  enter  into  the 
region  under  analysis.  Once  these  obstructions  have  been  found  and  identified,  (from 
the  conceptual  description  of  the  first  image  that  is  provided),  their  edge  orientations 
can  be  computed  and  corresponding  values  for  the  strength  function  reduced  at  those 
points.  Thus,  these  objects  do  not  interfere  with  the  essential  trace  computation  as 
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long  as  their  edges  are  not  coincident  and  in  the  same  orientation  as  those  of  the  ob¬ 
ject  being  searched  for. 

When  the  relaxation  process  has  terminated,  the  established  edge  labels  are 
linked  into  a  segment  of  the  outline,  beginning  with  the  specified  starting  point.  The 
segment  must  then  be  verified  as  a  legitimate  part  of  the  outline.  If  a  continuous  (no 
gaps)  edge  entirely  within  the  region  is  found,  it  is  accepted.  If  the  edge  has  gaps,  and 
the  gaps  can  be  filled  by  inserting  edge  elements  that  satisfy  good  edge  continuation 
criteria,  and  are  completely  within  the  region,  then  the  section  is  corrected  and  ac¬ 
cepted.  Such  "filling  in”  processes  are  useful  for  poor  quality  images,  e.g.,  those  aris¬ 
ing  from  poor  dye  mixing.  In  other  cases,  a  check  is  first  made  to  see  if  any  of  the  ob¬ 
structing  structures  outlined  in  the  first  image  lie  in  the  current  search  region.  If  so, 
their  location  is  passed  down  to  the  relaxation  procedure  so  that  the  strength  function 
can  be  modified  and  the  process  repeated.  Otherwise,  the  region  size  is  increased  on 
the  assumption  that  the  prediction  did  not  fortell  the  observed  circumstances.  (The 
hypotheses  did  not  include  the  one  which  expected  the  motion  which  was  actually  ob¬ 
served).  Finally,  if  no  outline  can  be  found,  the  prediction  can  be  taken  as  the  outline. 

When  an  outline  segment  has  been  verified,  the  next  one  is  considered.  It  is 
overlapped  with  the  previous  one  by  at  least  one  point  which  has  certainty  1.0  for  its 
appropriate  edge  label.  This  anchoring  provides  a  guarantee  of  good  continuation 
between  segments.  However,  it  is  attenuated  over  distance.  Thus,  the  segments  should 
be  kept  short. 

An  example  of  the  prediction-guided  relaxation  labelling  process  is  shown  in 
Fig.  6.2.  The  raw  image  is  shown  in  (a),  while  the  prediction  window  is  shown  in  (b). 
Figure  6.2(c)  shows  the  best  edge  using  the  output  of  a  5X5  gradient  operator  alone. 
The  best  response  at  each  pixel  is  selected  and  then  edge  continuity  criteria  are  ap¬ 
plied  in  order  to  determine  semantically  valid  edge  points.  These  edge  points  are  then 
linked  using  straight  line  segments.  Note  that  no  valid  edge  points  were  found  in  the 
upper  right  hand  side  of  the  image.  Figure  6.2(d)  shows  the  resultant  outline  using  the 
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FIGURE  6.2(a)  THE  ORIGINAL  IMAGE 
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FIGURE  6.2(b)  THE  PREDICTION  WINDOW 
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FIGURE  6.2(c)  BEST  EDGE  DETERMINED  BY  A  5  X  5  GRADIENT  EDGE  OPERATOR 
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FIGURE  6.2(d)  BEST  EDGE  USING  LOCATION  AND  ORIENTATION  BIASED  RELAXATION  LABELLING  PROCESS 
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FIGURE  6.2(e)  BEST  EDGE  USING  LOCATION  BIASED  RELAXATION  LABELLING  PROCESS 


location  and  orientation  biased  relaxation  labelling  process.  The  best  edge  is  found  for 
each  sub-region  as  in  Figure  6.2(c).  However,  in  this  case,  the  number  of  erroneous 
edge  points  is  kept  track  of  and  if  large  (say,  3  or  4),  relaxation  is  applied  to  the  points 
in  the  sub-region.  Out  of  24  segments  in  the  prediction  window,  9  did  not  require  relax¬ 
ation  at  all,  while  5  required  the  maximum  number  of  iterations  (5)  imposed  as  a  sys¬ 
tem  constraint.  In  total,  49  iterations  were  required  over  the  24  segments.  The  outline 
in  the  figure  is  essentially  the  same  as  that  which  cardiologists  use  in  their  diagnoses. 
In  comparison,  Figure  6.2(e)  shows  the  outline  determined  using  only  location  guided 
relaxation.  Note  that  in  a  large  portion  of  the  outline,  no  valid  edge  points  were  found, 
and  that  61  iterations  were  required  in  total  for  a  grossly  inferior  resultant  outline. 

6.2. 1.2  Descriptors  for  the  Raw  Trace 

The  raw  outline  is  used  to  determine  size,  location  and  shape  information. 
Shape  features  are  basically  convex  and  concave  sections.  There  is  a  need  for  descrip¬ 
tions  that  form  a  bridge  between  quantitative  and  qualitative  descriptions,  and  that 
lend  themselves  naturally  to  higher  level  abstractions.  Such  qualitative  descriptors  ap¬ 
pear  in  the  geography  of  coastlines  [Moore54].  Therefore,  the  particular  types  of  con¬ 
vexity  and  concavity  will  be  classified  using  terms  found  in  geography.  Examples  of  the 
2D  shape  features  are  lakes,  islands,  bays,  capes,  inlets,  coves,  fiords,  lagoons  of  all 
types,  concave  and  convex  points,  peninsulae,  hooks,  etc.  Each  has  associated  descrip¬ 
tors:  centroid,  area,  arclength,  longitudinal  axis,  lateral  axis,  and  maximum  interior  di¬ 
mension. 

The  algorithm  for  the  determination  of  bays  and  capes  is  as  follows.  The  out¬ 
line  is  traversed  in  a  counter-clockwise  direction  and  significant  left  and  right  turns 
are  detected.  A  bay  is  defined  as  a  right  turn,  followed  by  a  sequence  of  straight  seg¬ 
ments  and  left  turns,  and  ending  in  another  right  turn.  Similarly,  a  cape  is  a  left  turn, 
followed  by  straight  segments  and  right  turns  and  ending  with  a  left  turn.  Relative 
lengths  of  axes  and  maximum  interior  dimensions  then  are  used  to  further  classify  the 
shape  type.  For  example,  a  fiord  is  a  bay  whose  lateral  axis  is  much  longer  than  its 
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longitudinal  axis,  and  this  same  definition  applies  for  a  peninsula  as  a  sub-class  of 
capes.  Smoothing  of  the  outline  is  accomplished  by  running  the  shape  determination 
algorithm  on  the  rough  outline,  and  then  removing  any  shape  features  (replacing  them 
with  straight  line  segments)  whose  arclength  is  smaller  than  some  system  resolution. 
Such  a  scheme  for  shape  description  of  a  closed  outline  is  similar  to  that  described  in 
[Pavlidis79a]. 

Shape  information  may  be  hierarchically  described  if  the  domain  requires  the 
added  level  of  complexity  for  the  shapes  it  includes.  In  arriving  at  the  shape  hierar¬ 
chies,  small  shape  features  are  smoothed  out  and  the  shape  analysis  is  run  again.  Posi¬ 
tional  information  of  the  shape  features  in  both  the  gross  and  detailed  levels  of 
description  then  determines  which  detailed  features  are  encompassed  by  a  gross 
feature. 

Size  and  location  are  straightforward.  Segments  of  the  outline  have  points 
with  (x,y)  co-ordinates  and  an  arclength  while  objects  have  a  centroid,  arclength  (per¬ 
imeter),  area,  longitudinal  and  lateral  axes.  An  example  of  an  essential  trace  for  an 
object  is  given  in  chapter  7. 

6.2.2  Computing  the  Essential  Kineses 

Computation  of  the  inter-image  description  for  an  object  involves  comparison 
of  two  consecutive  essential  traces  for  the  object.  Motion  is  detected  for  the  object  if 
the  locations  of  any  of  its  parts  differ  between  the  two  traces.  Such  small  changes  may 
be  due  to  noise  or  to  quantization  effects.  However,  at  this  level  all  changes  are  as¬ 
sumed  to  be  legitimate  and  it  is  up  to  the  hypothesis  matching  and  rating  schemes  to 
weed  out  the  artifacts  from  the  real  changes.  As  discussed  in  Chapter  4,  the  lowest  lev¬ 
el,  or  primitive,  frame  descriptions  are  those  for  location  changes,  area  changes,  shape 
changes,  length  changes  and  time  intervals.  The  changes  detected  between  two  con¬ 
secutive  images  will  be  described  in  terms  of  these  frame  types.  The  set  of  such  frame 
tokens  for  a  particular  object  will  be  refered  to  as  the  essential  kineses  for  that  ob¬ 
ject.  In  other  words,  the  essential  kineses  for  an  object  are  the  changes  of  its  physical 
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properties. 

6.2.2. 1  Correlation  of  Shape  Features 

For  rigid  objects,  the  correspondence  between  components  in  two  successive 
images  can  be  computed  by  minimizing  global  distance  criteria  [Ullman79].  However, 
the  problem  domain  of  LV  wall  motion  involves  non-rigid  object  motion,  thus  making 
the  correspondence  much  more  involved.  Fig.  6.3  contains  examples  of  different  kinds 
of  superimposed  LV  outlines  that  indicate  the  types  of  motions  involved.  In  particular, 
problems  arise  because: 

•  The  number  of  shape  features  along  the  wall  is  not  constant.  There  are  a  small 
number  (  10  -  20  )  in  each  outline  and  the  number  may  vary  by  a  few  from  image  to 
image. 

•  Shape  features  may  protrude  from  other  larger  features  and  may  also  disappear 
into  other  larger  features  between  images. 

•  Continuity  criteria  must  be  satisfied  at  the  boundaries  between  segments. 

•  Shape  features  do  not  move  rigidly  -  they  stretch,  distort,  translate,  and  rotate 
(in  2D). 

In  order  to  cope  with  these  problems,  several  different  kinds  of  information 
must  be  used.  Furthermore,  since  the  problems  exist  at  different  representational  lev¬ 
els,  we  have  been  unable  to  formulate  a  single  criterion  that  encompasses  all  of  them. 
Rather,  we  have  found  that  the  following  set  of  matching  heuristics  works  well  in  prac¬ 
tice.  These  heuristics,  together  with  their  underlying  rationale,  are  given  below. 

•  2D  distance  is  clearly  an  important  criterion  for  non-rigid  object  motion  as  well 
as  for  rigid  object  motion.  However,  relative  positions  of  start  and  end  points  of  the 
shape  features  along  the  outline  have  turned  out,  in  practice,  to  be  much  more  power¬ 
ful.  Both  are  used. 

•  Between  images,  no  point  on  the  wall  moves  very  far.  The  data  collection  system 
we  have  implemented  [Delgrande79]  is  being  used  to  determine  values  for  the  average 
amounts  of  motion  for  the  different  wall  segments  in  normal  and  abnormal  situations. 
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NOTE:  Three  examples  of  LV  wall  motion 

patterns  are  shown.  The>  are  all  taken  in  the  RAO 
(Right  Anterior  Oblique)  projection.  Motion  of  individual  points 
on  the  LV  wall  can  also  be  seen.  These  motions  are  derived  from 
implanted  myocardial  screws. 

Both  a  and  b  are  patterns  from  patients  whose  LV's  are  functioning 
normally,  b  shows  the  "classic”  normal  contraction  pattern,  while 
A'  shows  a  different,  but  functionally  normal  pattern.  c  shows 
the  pattern  of  a  LV  with  mitral  stenosis. 


FIGURE  6.3  EXAMPLES  OF  LV  CONTRACTION  PATTERNS 
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•  Similarity  of  shape  type  is  a  different  kind  of  heuristic.  Shape  types  seem  to 
remain  constant,  with  "continuous"  changes  between  types.  For  example,  a  deep  bay 
cannot  become  a  deep  peninsula  between  images.  It  must  first  become  a  small  bay,  a 
straight  section,  a  small  cape,  and  then  a  peninsula.  Also,  shape  features  which  pro¬ 
trude  or  are  absorbed  from  one  image  to  the  next  usually  have  correspondence  map¬ 
pings  in  which  one  shape  feature  becomes  three  (it  "buckles")  or  vice  versa.  Explicit 
knowledge  of  such  transitions  greatly  facilitates  matchings  between  new  shape  features 
and  current  ones.  Other  more  complex  mappings  seem  to  be  much  less  frequent, 
which  makes  the  embedding  of  these  transitions  into  matching  heuristics  very  useful  . 

•  Complex  shape  feature  matchings  as  described  above  are  dependent  on  shape 
arclength  and  relative  positions  of  the  start  and  end  points  of  the  shape  along  the  out¬ 
line. 

•  Since  shape  types  overlap,  (a  bay  and  an  adjacent  cape  have  common  seg¬ 
ments),  matching  of  one  of  these  features  leads  to  partial  matches  for  the  adjacent 
overlapping  ones. 

•  If  more  than  one  level  of  shape  description  is  present,  the  matching  can  proceed 
in  a  top-down  manner,  from  the  more  gross  features  to  the  more  detailed  features. 

These  heuristics  guide  a  matching  procedure  that  attempts  to  find  the  best 
match  for  each  shape  feature  between  the  two  images.  For  each  shape  feature  in  the 
first  image,  candidates  for  matching  are  found  using  2D  distance  and  relative  location 
on  the  outline  criteria.  Similarity  of  shape  type  and  continuity  of  location  and  size  are 
then  used  to  determine  whether  or  not  one  of  the  candidates  can  match  the  shape 
feature  under  consideration.  If  no  single  candidate  is  applicable,  groups  of  shape 
features  are  considered  depending  on  the  size  of  the  shape  feature.  The  heuristics  are 
applied  again  in  order  to  find  matchings  of  the  one  shape  feature  into  several  others 
(i.e.,  it  has  "warped").  Similarily,  depending  on  feature  sizes,  matchings  of  several 
shape  features  into  a  single  one  can  be  found  (i.e.,  they  have  "straightened"  out).  Fi¬ 
nally,  if  no  clear  matches  of  the  above  types  are  found,  groups  of  shapes  are  compared 
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to  groups  of  shapes  In  each  image  again  using  size  and  location  criteria.  Matches,  once 
determined,  can  aid  in  the  discovery  of  successive  matches  using  the  last  two  of  the 
heuristics  listed  above. 

There  is  no  need  to  consider  the  possible  start  points  for  the  matching,  be¬ 
cause  all  traces  are  structured  in  the  same  manner  (they  are  all  derived  from  the 
predictions).  An  example  of  the  result  of  these  correspondences  is  shown  in  Fig  6.4. 
Two  consecutive  outlines  are  shown,  along  with  the  shape  features  for  each,  and  the 
correlation  between  them  determined  by  the  heuristics  described  above. 

The  raw  kineses  of  an  object  are  a  summary  of  its  movements  over  time. 
They  are  obtained  by  matching  the  feature  descriptors  of  the  object  using  the  six 
classes  of  heuristics  that  were  listed  above.  They  are  described  using  the  primitives 
defined  in  Chapter  4;  they  are  more  fully  described  in  the  next  section. 

The  matching  is  similar  to  that  found  in  [Pavlidis79b]  in  that  high  level  struc¬ 
tures  are  used  in  order  to  simplify  the  matching  procedure.  Also  Davis  [Davis77]  has  a 
matching  procedure  which  operates  on  the  same  images  as  Pavlidis’  except  that  relax¬ 
ation  labelling  methods  are  used.  These  two  efforts  demonstrate  that  the  shape 
matching  problem  can  be  successfully  tackled  in  a  number  of  ways.  Such  high  level 
descriptions  and  matching  procedures  have  several  advantages:  the  descriptions  are 
orientation  independent  because  each  shape  type  can  be  described  in  terms  of  its 
neighbours;  all  the  properties  of  the  structures  are  used  in  the  matching  -  size,  shape 
type,  orientation;  and,  the  matching  process  is  simplified  and  is  intuitively  clear  in 
description. 

6. 2. 2. 2  The  Trans formation  Primitives 

The  second  step  in  the  process  is  the  abstraction  of  primitive  motion  descrip¬ 
tions  from  these  correspondences.  Five  primitives,  as  described  in  Chapter  4,  are  pro¬ 
vided  from  which  all  higher  level  descriptions  are  obtained:  TIME_INTERVAL  -  the  time 

interval  over  which  changes  occur;  LOCATION _ CHANGE  -  change  of  location  of  a  point; 

AREA — CHANGE  -  change  of  the  area  of  the  region  representing  an  object; 
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FIGURE  6.4  AN  EXAMPLE  OF  SHAPE  MATCHING 


outline  1  outline  2 

The  above  shows  two  consecutive  LV  outlines,  with  point  correspondencies  labelled 
as  determined  by  shape  feature  matching.  Notice  that  points  12',  13',  14',  and  15 
show  an  example  of  the  "buckle"  transformation  primitive,  where  the  "cape"  formed 
them  protrudes  from  the  "lagoon"  formed  by  points  1  through  4  in  outline  1. 
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LENGTH_CHANGE  -  change  of  the  length  of  either  an  axis  or  the  arclength  of  an  object; 

and  SHAPE_CHANGE  -  change  of  the  shape  of  an  object.  The  latter  three  are  all  IS _ A 

descendents  of  PHYS_PROP_CHANGE. 

The  semantic  components  of  LOCATION_ CHANGE  are  a  subject  which  exhibits 
the  motion,  a  start  and  an  end  time,  duration  ,  speed,  and  trajectory.  Trajectory  has 
components  in  the  same  directions  as  the  subject’s  longitudinal  and  lateral  axes,  rath¬ 
er  than  in  the  horizontal  and  vertical  directions.  The  intent  here  is  to  provide  a  set  of 
primitive  motion  descriptors  which  can  be  used  by  higher  levels  of  the  system  to  pro¬ 
duce  more  specific  motion  descriptions.  It  is  very  difficult  to  relate  x-y  stretch,  etc.,  to 
higher  level  natural  language  verbs  without  constantly  changing  the  co-ordinate  sys¬ 
tem.  However,  longitudinal  and  lateral  directions  relate  directly  to  higher  level  verbs. 

LOCATION_CHANGE  leads  directly  to  the  two  predicates  at  the  next  higher  lev¬ 
el  of  description:  TRANSLATE  and  ROTATE.  TRANSLATE  is  used  if  the  subject’s  centroid 
changes  location,  and  ROTATE  if  the  subject’s  axes  change  orientation.  Rotations  about 
exterior  points  are  handled  by  a  verb  such  as  GO_AROUND. 

The  semantic  components  of  PHYS_PROP_CHANGE  are  a  subject  whose  pro¬ 
perty  is  changing,  the  property  that  changes,  temporal  relations,  (i.e.,  start,  end  times 
and  duration)  and  a  rate  of  change.  Depending  on  the  property,  one  of  several  more 
specific  PHYS_PROP_CHANGE’s  may  be  used.  SHAPE_CHANGE  is  used  for  major  shape 
changes.  It  is  also  classified  further  into:  DISTORT  -  changes  in  relative  position  of  an 
object’s  axes;  BUCKLE  -  for  l-to-3  shape  feature  mappings,  STRAIGHTEN  -  for  n-to-1 
shape  feature  mappings;  and,  WARP  for  all  other  shape  feature  mappings. 
LENGTH-CHANGE  can  be  either  a  WIDEN,  COMPRESS,  NARROW,  or  a  LENGTHEN. 
AREA__CHANGE’s  are  CONTRACT’S  or  EXPAND’s. 

For  example,  the  frame  tokens  which  would  define  the  changes  in  the  LV  as  a 
whole  between  two  consecutive  images  are: 

token  lv_motion _ 1  instance _ of  PHYS_PROP_CHANGE  with 

centroid _ change  :  cent_loc_change_l  ; 

long_axis_change  :  stretch_l  ; 
lat_axis_change  :  stretch _ 2  ; 
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area _ change  :  area _ change_l  ; 

end 

token  cent_loc_change_l  instance_of  LOCATION_CHANTGE  with 
subj  :  lv_l  ; 

time_int  :  ti _ 1  ; 

start _ loc  :  xy _ coords_l  ; 

end _ loc  :  xy_coords_2  ; 

end 

The  remainder  of  the  tokens  stretch_l,  stretch _ 2,  etc.  are  defined  in  analogous  ways. 

Frame  tokens  such  as  these  are  provided  not  only  for  the  LV  as  a  whole,  but  also  for 
each  of  its  parts  (LV  wall,  its  segments,  mitral  valve,  etc.),  in  accordance  with  the  LV 
PART_OF  object  hierarchy. 

The  set  of  primitive  motions  associated  with  an  object  between  two  consecu¬ 
tive  images  at  all  levels  of  its  PART_OF  hierarchy  defines  the  essential  kineses  for  that 
object.  The  notion  of  using  primitives  for  inter-image  description  of  motions  has  also 
been  investigated  by  O'Rourke  [0'Rourke79],  He  uses  primitives  such  as  TWIST,  RO¬ 
TATE,  and  MOVE  along  with  their  appropriate  semantic  components  in  order  to 
describe  the  inter-image  changes  in  films  of  human  motion. 

6.3  Matching  -  Comparing  the  Data  to  Hypotheses 

The  previous  section  described  how  the  lowest  level  descriptions  of  the  ob¬ 
jects  and  their  motions  are  computed.  These  descriptions  may  or  may  not  be  directly 
comparable  to  those  which  are  present  in  the  currently  hypothesized  frames.  There¬ 
fore  there  are  two  distinct  steps  which  need  to  be  performed. 

The  first  step  determines  whether  or  not  the  description  level  in  the  hy¬ 
pothesis  to  be  matched  against  is  the  same  as  that  of  the  description  derived  from  the 
observations.  This  can  be  done  by  noting  the  level  of  the  descriptive  terms  used  in  the 
hypothesis  (their  level  in  the  1S_A  hierarchy).  If  they  are  not  at  the  same  level  as  the 
essential  kineses,  then  a  bottom-up  re-computation  of  descriptions  must  be  done.  In 
fact,  the  IS_A  hierarchy  is  searched  downwards  from  the  concepts  used  in  the  essential 
kineses,  matching  against  their  "son"  frames  in  order  to  determine  which  of  them  is 
applicable.  This  proceeds  until  the  proper  level  is  reached.  For  example,  if,  at  the 
lowest  level,  the  descriptive  term  is  AREA _ CHANGE,  and  the  hypothesis  to  be  matched 
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against  uses  CONTRACT,  then  each  frame  below  AREA _ CHANGE  is  checked  in  order  to 

see  if  CONTRACT  lies  below  it.  If  it  does  ,  the  specified  constraints  are  checked  and,  if 
satisfied,  the  CONTRACT  term  is  added  to  the  description.  Failure  at  this  point  indi¬ 
cates  failure  of  the  match  and  the  proper  exception  is  raised. 

As  a  more  complex  example,  consider  the  frame  for  ISCH_CAP_ISO_RELAX 
presented  in  Appendix  A.  In  this  frame,  the  motion  of  the  apical  segment  of  the  left 
ventricle  is  described  in  terms  of  the  directionals  TOWARDS  and  AWAY_FROM  in  con¬ 
junction  with  specific  instances  of  TRANSLATE’S.  The  essential  kineses  for  the  segment 
will  be  in  terms  of  L0CAT10N_CHANGE’s.  We  know  from  the  generic  motion  KB  that 
TRANSLATE  1S_A  LOCATION_CHANGE  and  that  TOWARDS  and  AWAY_FR0M  IS_A 

OBJ_TRANSLATE,  and  03J_TRANSLATE  IS _ A  TRANSLATE.  Thus,  in  order  to  derive  a 

description  of  the  segment’s  motion  in  the  same  terms  as  in  the  frame,  the  constraints 
in  the  TRANSLATE  frame  are  checked  against  the  data  from  the  L0CATI0N_C RANGE  to¬ 
ken  provided  by  the  essential  kineses.  The  same  is  done  for  the  TOWARDS, 
AWAY_FR0M,  and  OBJ_TRAN SLATE  frames.  If  the  constraints  are  satisfied,  then  in¬ 
stances  of  these  frames  are  added  to  the  instance  KB  and  matching  succeeds. 

Once  the  description  levels  match,  the  concepts  and  constraints  are  checked 
one  by  one  -  each  failure  raising  the  required  exception,  including  the  one  which  pro¬ 
pagates  up  the  frame’s  PART_OF  hierarchy.  This  produces  a  (possibly  null)  set  of  ex¬ 
ception  tokens  for  the  hypothesis  which  was  being  matched.  Exception  frames  have 
been  fully  discussed  in  previous  chapters. 

6.4  Change  of  Attention  Mechanism 

Let  us,  for  a  moment,  consider  the  change  of  attention  mechanisms  in  two 
other  large  application  systems  KEARSAY-II  [Lesser77]  and  INTERNIST-II  [Pople77].  In 
HEARSAY-II,  knowledge  sources  are  activated  whenever  they  see  their  prerequisite  data 
on  the  blackboard,  i.e.,  the  activation  of  knowledge  sources  is  data-driven.  In 
INTERNIST-II,  hypotheses  are  also  generated  from  the  data.  Signs  and  symptoms  are 
input  to  the  system  and  the  appropriate  hypotheses  are  activated.  Here,  data  is  con- 
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tinually  being  added  as  a  result  of  questions  asked  by  the  system,  so  that  new  hy¬ 
potheses  are  introduced  throughout  the  diagnostic  process. 

Our  change  of  attention  mechanism  differs  in  that  it  is  failure  driven  rather 
than  success  driven.  That  is,  matching  successes  do  not  cause  new  hypotheses  to  be 
introduced:  matching  failures,  on  the  other  hand  do.  Our  system  maintains  a  set  of  ac¬ 
tive  hypotheses  throughout  the  recognition  process.  This  set  is  initially  filled  with  the 
motion  hypotheses  that  the  user  believes  may  be  exhibited  by  a  particular  object  (or 
objects). 

When  a  motion  hypothesis  is  activated,  several  others  are  also  automatically 
activated: 

•  any  hypotheses  which  are  its  IS _ A  ancestors.  The  reason  for  this  can  be  seen  by 

considering  the  definition  of  IS_A.  Properties  of  one  frame  are  inherited  by  its  IS _ A 

sons.  Therefore,  through  inheritance,  different  levels  of  description  are  allowed  to  co¬ 
operate,  providing  both  the  coarse  levels  of  description,  that  aid  in  error  suppression 
through  feedback  and  the  detailed  descriptions,  which  are  necessary  for  discrimination 
between  concepts. 

•  any  hypotheses  for  the  current  time  interval  which  form  component  motions  of 
the  hypothesis.  These  are  determined  through  the  PART_OF  hierarchy  for  the  frame, 
i.e.,  its  slots. 

This  definition  is  applied  recursively. 

Active  hypotheses  are  organized  by  their  "conceptual  adjacency".  This  may 
be  likened  to  the  "structural  adjacency"  of  HEARSAY-II,  which  is  based  on  immediate 
adjacency  in  the  AND/OR  tree  of  hypotheses.  Our  scheme,  however,  is  more  strongly 
related  to  the  semantics  of  the  hypotheses  in  question.  If  two  hypotheses  define  mo¬ 
tions  for  the  same  subject  and  for  the  same  time  interval,  then  we  say  that  they  satisfy 
the  common  subject-time  condition  (CST).  This  concept  is  used  in  the  following  dis¬ 
cussion.  By  definition,  two  hypotheses  H  and  H'  are  adjacent  conceptually,  if  one  (or 
more)  of  the  following  hold: 
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1 )  There  is  an  active  similarity  link  between  H  and  H\  and  H  and  K’  satisfy  the 
CST  condition.  Such  an  adjacency  implies  mutual  exclusion,  i.e.,  only  one  of  the  hy¬ 
potheses  can  be  instantiated. 

2)  H  preceeds  H’  in  time  or  H  follows  H’  in  time,  and  they  define  motions  for  the 
same  motion  subjects.  In  this  case,  there  is  overlap  of  time  interval  and  since  only  one 
motion  can  exist  during  any  interval,  the  hypotheses  are  competitors.  The  same  prob¬ 
lem  exists  in  speech  understanding  systems  at  word  boundaries. 

3)  H  IS_A  H’  and  CST  holds. 

4)  H  has  H’  as  a  direct  IS _ A  descendent,  and  the  CST  condition  holds. 

5)  H  is  adjacent  to  itself  for  the  next  time  frame.  This  handles  exceptions  in  the 
parts  of  hypothesis  H  over  time.  Thus  the  PART_OF  hierarchy  is  implicitly  taken  into 
account. 

These  adjacency  types  encompass  all  of  the  organizational  axes  of  the 
knowledge  base.  It  is  not  possible  for  type  1  to  conflict  with  types  3  and  4  because  ac¬ 
tive  similarity  links  only  exist  between  frames  at  the  same  level  of  abstraction  (same 
level  in  the  IS_A  hierarchy).  Types  3  and  4  are  inverses  and  therefore  always  co-exist. 
This  co-existence  allows  enforcement  of  global  constraints  on  specific  concepts  as  well 
as  providing  assistance  in  the  instantiation  of  general  concepts  from  specific  concepts. 
Type  5  adjacencies  exist  for  every  active  hypothesis  and  these  cannot  conflict  with  any 
other  adjacency  since  it  is  a  unary  adjacency  and  all  the  others  are  binary.  Type  1 
cannot  conflict  with  type  2  because  of  the  CST  condition.  However,  there  is  a  default 
mechanism  that  converts  certain  kinds  of  type  1  adjacencies  to  type  2  This  mechan¬ 
ism  will  be  described  shortly.  There  are  certain  cases  when  type  2  may  conflict  with 
type  3  and  thus,  4  as  well.  Such  cases  arise  in  situations  where  K  preceeds  H’  in  time 
and  H  IS_A  H'.  This  could  easily  be  true  if  there  is  a  lack  of  information  about  the 
specifics  of  a  particular  motion  during  a  particular  time  interval.  For  example,  sup¬ 
pose  a  contraction  of  very  specific  nature  is  described  by  H,  with  specific  area  changes 
given.  For  the  next  time  interval,  the  only  information  known  is  that  the  contraction 
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continues,  without  any  particular  rates  of  area  changes  given.  Clearly,  the  first 
description  is  a  specific  version  of  a  contract  and  is  therefore  an  IS _ A  son  of  the  CON¬ 

TRACT  frame.  However,  they  are  also  temporally  adjacent.  In  such  cases,  types  3  and 
4  take  precedence  over  type  2  for  that  overlapping  time  interval.  If  a  hypothesis  is 

made  to  compete  with  an  IS _ A  ancestor  inconsistencies  would  be  introduced  into  the 

rating  process  because  during  the  time  spanned  by  both  H  and  H*.  H’  must  be  true  if  H 
IS_A  H’.  In  such  a  case,  the  5  adjacency  of  hypothesis  K  would  reduce  its  certainty 
factor  once  the  event  stops.  The  end  time  of  H  would  be  determined  by  the  temporal 
segmentation  process  discussed  in  a  later  section. 

An  object’s  set  of  possible  hypotheses  consists  of  all  hypotheses  which  define 
motion  classes  for  that  object  during  a  common  time  interval.  Of  those  which  are  1S_A 

related,  the  most  specific  is  always  considered  first  (followed  by  its  IS _ A  ancestor,  etc. 

-  like  a  stack  of  IS _ A  concepts).  Hypotheses  for  an  object’s  parts  are  considered  as 

part  of  the  set  for  that  particular  part.  In  this  way,  since  hypotheses  are  organized  by 
object,  there  can  be  no  overlapping  sets  of  hypotheses  -  no  one  hypothesis  can  be  a 
member  of  more  than  one  set.  There  is  no  restriction,  however,  on  the  PART_0F  rela¬ 
tions  between  sets  of  hypotheses. 

The  system  adds  hypotheses  to  its  list  of  active  ones  via  activated  similarity 
links  and  via  the  "next"  temporal  constraint,  (i.e.,  if  motion  X  is  the  "next"  one  after  Y, 
motion  Y  has  an  end  time  which  is  the  same  as  motion  X’s  start  time). 

A  similarity  link  in  hypothesis  H  is  activated  between  hypotheses  H  and  H’ 

when: 

•  the  similarity  expression  is  satisfied,  i.e.,  all  properties  of  K’  which  must  be  true 
before  H’  can  be  activated  must  have  been  instantiated:  and, 

•  at  least  one  of  the  exceptions  in  the  difference  expression  has  been  instantiated. 
It  is  not  necessary  that  this  be  the  first  one  in  the  time  course  expression  .  Noise 
effects  may  mask  it.  One  must  realize,  however,  that  noise  or  extraneous  information 
may  erroneously  trigger  the  similarity. 
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When  frame  H’  is  activated  by  H  as  its  competitor,  the  slots  of  H’  must  be 
filled  up  to  the  time  instant  of  activation.  It  will  be  true,  in  many  cases,  that  some  of 
the  slots  of  H  will  be  identical  in  content  to  those  of  H'.  These  are  transferred  through 
the  binding  expression  in  the  similarity  link.  Those  that  are  not  must  be  computed. 

There  are  several  additional  considerations  that  arise  when  determining  simi¬ 
larities  between  hypotheses: 

•  When  a  particular  exception  cannot  be  handled  by  the  local  similarity  links,  the 

links  of  its  IS _ A  ancestors  are  checked. 

•  Since  activation  of  a  frame  automatically  activates  its  IS _ A.  ancestors,  transfer 

of  parts  between  the  two  frames’  IS _ A  ancestors  may  be  accomplished  by  the  binding 

expressions  of  the  similarity  links  (if  present)  between  those  frames. 

•  There  is  also  a  need  to  propagate  similarities  upwards  along  the  PART_OF 
hierarchy,  because  possible  mis-matches  of  a  motion  component  may  require  com¬ 
pletely  new  contexts  for  the  newly  activated  parts.  This  is  done  by  automatically  rais¬ 
ing  an  exception  in  the  parent  hypothesis.  When  an  exception  is  detected  in  a  part  A  of 
frame  B,  this  automatically  generates  an  additional  exception  for  frame  B  stating  that 
frame  B  has  failed.  In  this  way  exceptions  are  propagated  up  the  PART_OF  hierarchy. 
The  exception  carries  with  it  a  special  slot  "prereq_part"  which  specifies  the  newly  ac¬ 
tivated  frame  name  C.  The  parent  frame  of  B,  that  is  D,  can  use  its  similarity  links  with 
the  added  constraint  that  C  must  be  PART_OF  any  newly  activated  destination  frame  of 
D. 

The  start  time  of  the  activated  hypothesis  is  taken  to  be  the  instant  of  activa¬ 
tion  of  the  activating  hypothesis.  This  is  reasonable  because  if  it  were  true  that  the  ac¬ 
tivated  hypothesis  should  have  started  earlier,  then  a  similarity  link  from  a  previous 
hypothesis  in  time  for  that  object  should  have  activated  it.  If  it  should  start  later,  then 
the  two  hypotheses  should  be  related  by  the  "next”  time  constraint.  Such  considera¬ 
tions  are  true  only  if  both  of  the  hypotheses  specify  durations  or  start  and  end  times. 
If  they  do  not,  then  it  is  ambiguous  as  to  whether  the  motions  follow  one  another  or  are 


-  110  - 


competitors  for  the  same  time  interval.  If  a  hypothesis  A  activates  B  through  a  similar¬ 
ity  link  and  no  timing  information  is  given,  then  this  is  considered  to  be  a  ’'next"  con¬ 
straint.  The  description  produced  would  be  that  the  object  exhibits  motion  A  for  a 
duration  of  X  and  then  motion  B  for  a  duration  of  Y. 

Suppose  a  hypothesis  A  activates  two  or  more  hypotheses  via  similarity  links 
at  the  same  time  instant.  Also,  assume  that  there  is  no  time  information  specified. 
Then  frame  A  has  a  "next”  relation  with  each  of  the  newly  activated  ones,  while  the  new 
ones  are  in  competition  with  one  another  for  that  time  interval.  If  time  information  is 
provided,  then  the  frames  are  all  similar  to  one  another. 

The  "next"  temporal  constraint  is  used  in  the  following  way.  When  the  current 
motion  hypothesis  reaches  its  specified  minimum  end  time,  its  "next"  motion  hy¬ 
pothesis  for  the  same  subject  is  activated.  The  focus  of  attention  mechanism  then  aids 
in  the  segmentation  between  the  two  hypotheses  in  order  to  find  their  time  boundary. 
For  example,  consider  the  N_LV_CYCLE  frame  in  chapter  5.  In  particular,  the  slots 
"systole"  and  "diastole"  can  illustrate  this  point  well.  These  phases  are  adjacent  in 
time.  From  their  respective  frames,  we  see  that  "systole"  ends  ,  at  its  latest,  at  0.38 
seconds  after  the  start  of  the  cycle,  while  "diastole"  begins  at  its  earliest,  0.36  seconds 
after  the  start  of  the  cycle  -  an  overlap  of  0.02  seconds.  Therefore,  the  N_DIAST0LE 
frame  would  be  activated  at  0.36  seconds  into  the  cycle,  while  the  N_SYST0LE  frame 
would  be  de-activated  at  0.38  seconds  into  the  cycle.  The  changes  in  certainty  factors 
for  these  two  frames  during  the  overlap  interval  would  then  determine  exactly  where 
their  time  boundary  would  be.  The  criteria  for  time  segmentation  are  discussed  in  sec¬ 
tion  6.6. 

Hypotheses  are  deleted  from  the  active  list  in  two  ways.  If  the  hypothesis’  ex¬ 
pected  maximum  duration  is  exceeded,  it  is  removed.  If  the  values  of  its  current  slots 
can  be  found,  it  is  instantiated  (added  to  the  description)  and  its  certainty  factor  com¬ 
puted  (see  next  section).  If  the  slots  cannot  be  filled  it  is  abandoned  completely.  The 
second  way  of  removing  hypotheses  is  by  thresholding.  Once  a  hypothesis’  certainty 
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falls  below  some  fraction  of  the  leading  hypothesis’  certainty,  it  is  deleted.  The  hy¬ 
potheses  must  of  course  satisfy  the  common  subject-time  condition.  As  we  shall  see  in 
the  following  section,  the  danger  of  removing  viable  hypotheses  is  greatly  reduced  be¬ 
cause  of  the  nature  of  the  certainty  factor  updating  mechanism.  Deletion  of  a  hy¬ 
pothesis  is  propagated  to  all  of  its  parts,  all  parent  subjects,  and  all  IS_A  sons  recur¬ 
sively.  Of  course,  if  a  part  or  IS _ A  ancestor  is  used  by  another  hypothesis  as  well,  it  is 

not  deleted. 

It  is  assumed  that  any  pair  of  hypotheses  which  are  mutually  exclusive  have  a 

similarity  link  between  them,  even  if  only  by  inheritance  through  an  IS _ A  ancestor.  It 

may  even  be  a  trivial  link,  such  as: 
similarity  links 

sim _ link_l  :  S 

for  differences  : 

dl  :  EXCEPTION  ; 

This  link  could  activate  frame  S  from  another  frame  if  any  exception  at  ail  is 
detected  Ln  that  other  frame’s  matching  progress.  As  a  matter  of  fact,  if  such  a  link  is 
placed  high  enough  in  the  IS_A  hierarchy  it  could  be  used  as  a  default  link  for  all  of 

that  node’s  IS _ A  descendents.  It  should  be  clear  in  this  case  however,  that  the  final 

level  of  description  will  be  at  a  level  no  more  specific  than  that  of  the  frame  in  which 
the  similarity  link  was  found.  There  is  no  mechanism  present  to  traverse  down  the 

IS _ A  hierarchy  in  order  to  find  a  more  specific  description. 

6.5  Focus  of  Attention  Mechanism 

A  focus  of  attention  mechanism  is  needed  because  the  prediction  mechanism 
must  know  which  are  the  "best”  hypotheses  at  any  instant  in  the  processing,  so  that  it 
can  base  its  expectations  on  them.  The  hypotheses  will  be  ordered  by  means  of  the 
certainty  that  the  system  has  in  them.  The  best  candidates  are  determined  for  each 
object;  that  is,  hypotheses  certainty  values  are  not  compared  unless  the  hypotheses 
define  motions  for  the  same  object  and  for  the  same  time  interval,  (the  common 
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subject-time  condition).  Thus,  each  object  would  have  a  leading  hypothesis,  and  each 
part  of  each  object  would  have  a  leading  hypothesis.  It  is  not  necessarily  true  that  the 
best  hypothesis  for  an  object's  part  is  PART_OF  related  to  the  best  hypothesis  for  the 
object  as  a  whole. 

Each  hypothesis  has  a  certainty  factor  associated  with  it  -  a  number  between 
0.0  (completely  uncertain)  and  1.0  (completely  certain).  Initial  values  are  set  through 
the  conceptual  adjacencies  of  each  hypothesis  when  it  is  activated.  If  a  hypothesis  H, 
has  N  competitors  (either  mutually  exclusive  ones,  through  similarity  links,  or  through 
the  "next”  time  constraint),  then  each  hypothesis  of  the  competing  set  has  an  initial 
certainty  of  1/N  *  cert(K).  In  this  way,  the  addition  of  hypotheses  via  a  similarity  link 
does  not  require  re-normalization  of  certainty  values  and  each  of  the  new  hypotheses 
has  an  equal  chance  of  succeeding.  It  would  also  be  possible  to  include  explicitly  with 
the  similarity  link  an  a  priori  initial  certainty  value  perhaps  dependent  on  the  condi¬ 
tions  verified  in  the  similarity  and  difference  expressions.  Such  a  mechanism  is  left  for 
future  work;  it  can  not  at  the  present  time  be  included  in  ALVEN  because  such  informa¬ 
tion  is  not  known  to  cardiologists.  If  a  hypothesis  has  no  competitors,  then  as  far  as 
the  system  is  concerned,  it  is  certain  that  the  hypothesis  will  be  instantiated  -  because 
there  is  no  other  possibility.  This  is  why  similarity  links  are  so  crucial  -  matching 
failures  would  not  be  recorded  in  a  hypothesis’  certainty  factor  through  the  relaxation 
process  unless  they  also  activate  a  similarity  link  to  a  competing  frame.  The  reason 
for  this  is  that  certainty  factors  are  normalized  over  the  set  of  competing  hypotheses. 
As  in  relaxation  labelling  for  edge  detection,  the  competing  labels  for  a  point  were  nor¬ 
malized  so  that  the  sum  of  their  certainties  remained  at  1.0.  Here  as  well,  the  compet¬ 
ing  labels  for  any  object’s  motion  are  normalized.  When  a  hypothesis  activates  a  simi¬ 
lar  frame,  they  share  equally  the  activating  hypothesis'  certainty  factor. 

Several  other  systems  use  similar  kinds  of  certainty  factors.  Medical  diag¬ 
nosis  systems  generally  make  heavy  use  of  such  numbers  [Shortliffe76],  [Szolovits79], 
etc.  In  such  systems,  initial  certainties  are  known  a  priori  from  detailed  studies  of  the 
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number  of  patients  suffering  from  particular  diseases  which  exhibit  certain  signs  and 
symptoms.  In  other  words,  they  are  part  of  the  knowledge  base  -  they  reflect  the 
current  best  knowledge  of  the  diagnostician  about  the  particular  disease.  The  scoring 
of  hypotheses  during  the  diagnostic  process  has  led  to  a  number  of  ad  hoc  schemes: 
they  involve  heuristic  focussing  and  avoid  formal  approaches.  The  reason  for  this  is 
that  discrimination  from  among  several  hypotheses  (differential  diagnosis)  and  the 
consideration  of  concurrent  hypotheses  present  difficult  problems.  The  approaches 
used  are  very  specialized  and  do  not  seem  to  be  generalizable  to  other  domains.  For 
example,  Bayesian  methods  require  a  priori  probabilities  for  the  domain.  Statistical 
pattern  recognition  involves  techniques  such  as  clustering  or  least  square  error- 
differences  between  observed  and  predicted  values.  Such  scoring  schemes  are  not 
semantically  related  to  the  domain  and  do  not  solve  the  problems  of  concurrent  or 
competing  hypotheses.  In  addition,  they  require  simplifying  assumptions  such  as  in¬ 
dependence  of  symptoms,  mutual  exclusion  of  diseases,  a  complete  knowledge  base  of 
diseases,  stability  of  symptoms  over  time  (that  is,  time  effects  are  not  handled  at  all). 

Our  certainty  factors  are  not  a  part  of  the  knowledge  of  the  domain  like  the  a 
priori  probabilities  are.  They  are  simply  a  device  used  for  rating  hypotheses  and  for 
determination  of  which  hypotheses  are  stronger  or  which  more  closely  describe  the 
conditions  observed  in  the  input  data:  a  "goodness  of  fit"  mechanism.  In  addition,  in 
the  medical  systems,  special  provisions  must  be  made  for  the  propagation  of  certainty 
factors  to  other  hypotheses  in  the  knowledge  base,  and  this  leads  to  quite  complicated 

mechanisms  such  as  the  one  in  MYCIN.  In  our  scheme,  propagation  is  an  inherent  pro- 

** 

perty  of  relaxation  labelling.  Hypotheses  receive  support  from  their  conceptual  neigh¬ 
bours:  a  change  in  a  neighbour  is  passed  on  to  any  other  hypotheses  for  which  it  is 
semantically  important. 

The  driving  concept  used  for  the  updating  of  certainty  factors  in  our  system 
is  generic  good  temporal  continuation  of  hypotheses.  This  is  direct  analogy  to  good 
edge  continuation  for  points  in  images.  Global  constraints  extracted  from  the 
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knowledge  organization  and  local  constraints  embodied  in  the  hypotheses  themselves 
provide  the  basis  for  the  determination  of  good  temporal  continuation. 

Once  the  initial  certainties  are  set  for  a  hypothesis,  the  certainty  is  updated 
for  each  subsequent  image.  The  updating  rule  that  we  use  is  taken  from  relaxation  la¬ 
belling  [Zucker77]  and  has  exactly  the  same  form  as  presented  in  that  paper.  The  only 
syntactic  change  is  that  we  need  to  keep  track  of  the  certainty  factors  at  each  time  in¬ 
stant  t.  Its  form  is  the  following: 


cert  (t+l.H)=~ 

noT~m  ( t.H  ) 

where  cert( t+  1  ,H)  is  the  certainty  factor  for  hypothesis  K  at  time  instant  ,  t  + 1, 
q(t,H)  is  the  contribution  from  its  neighbouring  hypotheses  (assume  that  there  are  n 
of  them),  at  time  t  and  is  given  by: 

q  ( t.H  )=  Y,  ™  ( H.Hi.t  )*comp  (. H.Hitt  )0cert  ( t.Hi ) 


i  =  1 


norm( t.H)  is  the  normalizing  factor  for  hypothesis  H.  It  is  given  by: 


■norm.  ( t.H  )=  £  cert  (t,Hi)0  [l  + q  (t,Hi)] 
i  =  1 

and  w  is  the  weighting  function  of  the  contribution  from  one  hypothesis  to  its  neigh¬ 
bour  at  a  particular  time  instant.  The  sum  of  the  contributions  must  be  unity.  Finally, 
comp  is  the  compatibility  function  between  two  hypotheses  at  a  particular  time  in¬ 
stant,  to  be  specified  below. 

The  major  change  from  the  process  described  by  Zucker  et  al.  is  that  we  do 
not  iterate  until  certainty  factors  converge  to  stable  values.  Rather,  certainties  are 
updated  using  only  one  application  of  the  rule.  In  this  way,  each  update  corresponds  to 
an  analyzed  image  in  the  sequence.  The  reason  for  this  change  requires  a  clear  under¬ 
standing  of  what  the  relaxation  labelling  process  (RLP)  does. 

For  edge  determination  in  single  images,  the  responses  of  an  edge  operator 
supply  the  initial  certainty  factors  for  the  possible  labels  for  each  point.  For  the  pur¬ 
poses  of  this  argument,  let  us  assume  that  there  are  three  labels.  This  initial  assign¬ 
ment  of  certainty  factors  place  the  starting  point  of  the  RLP  at  some  point,  say  A,  on 
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the  plane  formed  by  the  certainty  factor  vectors  [1,0,0$,  [0,1,0$,  [0,0,1$  -  see  Figure  6.5. 
The  plane  is  as  it  is  because  the  sum  of  the  certainty  factors  must  remain  at  one;  oth¬ 
erwise,  the  process  is  not  guaranteed  to  converge  .  From  point  A,  the  RLP,  using  good 
edge  continuity  criteria  takes  the  certainty  factor  vector  through  some  path  to  a  final 
convergent  labelling  as  shown  in  the  figure.  Suppose  that  in  our  motion  analysis  case, 
the  same  were  done  between  images.  The  effect  is  disastrous.  Because  of  the  nature  of 
the  RLP,  the  system  could  never  move  away  from  this  final  stable  state,  and  therefore 
after  the  first  pair  of  images,  the  certainty  factors  could  not  change.  This  is  clearly  un¬ 
desirable. 

In  Chapters  1  and  2  wre  motivated  the  need  for  a  focus  mechanism  based  on 
the  concept  of  inertia  for  motion  understanding.  This  concept  can  now  be  expanded 
upon.  If  only  one  application  of  the  updating  rule  is  used  between  images,  the  effect  is 
that  the  system  moves  along  a  short  vector  tangent  to  the  path  that  is  would  have  tak¬ 
en  if  many  iterations  were  done.  Two  images  are  enough  to  determine  this  vector.  The 
system  thus  has  inertia  of  location  because  it  does  not  move  very  far  away  from,  its 
current  location.  This  is  desirable  because  hypotheses  and  their  compatibility  values 
are  uncertain.  The  length  of  the  vector  is  determined  by  how  consistent  the  matchings 
of  hypotheses  and  the  kineses  are  with  the  constraints  imposed  by  the  organizational 
axes  of  the  KB.  The  system  also  has  inertia  of  direction  because  it  moves  along  a 
tangent  to  its  current  path  in  the  plane  formed  by  the  hypothesis  space.  The  problem 
of  rapidly  shifting  foci  is  therefore  greatly  reduced.  However,  other  considerations  ar¬ 
ise.  What  are  the  limitations  of  such  a  process  in  terms  of  how  many  iterations  (in  oth¬ 
er  words,  images)  are  needed  in  order  to  determine  a  consistent  labelling,  and  how  is 
such  a  process  affected  by  extraneous  stimuli?  Empirical  evidence  presented  in 
Chapter  7  answ-ers  these  questions  and  provides  support  for  our  design  decisions. 

The  compatibilities  between  hypotheses  depend  directly  on  the  types  of  con¬ 
ceptual  adjacencies  that  hypotheses  exhibit.  The  types  of  adjacencies  are  listed  below 
and  their  associated  compatibilities  are  discussed.  It  should  be  noted  that  the  compa- 
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tibilities  depend  not  only  on  the  conceptual  adjacency  between  the  hypotheses,  but 
also  on  the  matching  progress  of  the  adjacent  hypothesis. 

1)  As  described  in  the  previous  section,  this  type  of  adjacency  implies  mutual  ex¬ 
clusion  between  the  hypotheses  because  they  are  competitors.  If  H’  succeeds  in 
matching,  then  it  removes  support  from  H  while  if  it  fails  it  gives  support  to  H  . 

2)  The  two  hypotheses  are  competitors  for  their  overlapping  time  interval  and 
therefore  the  compatibility  is  set  in  this  case  so  that  the  time  intervals  occupied  by 
the  two  motions  may  be  segmented.  The  hypotheses  cannot  co-exist  in  the  same  time 
interval;  therefore,  they  are  mutually  exclusive  and  the  compatibilities  are  set  as  in 
1)- 

3)  If  H’  succeeds,  then  there  is  some  evidence  for  K,  its  IS _ A  son,  to  succeed.  H’ 

should  support  H  with  a  positive  compatibility.  The  particular  value,  however,  should 
be  domain  dependent.  For  example,  it  could  depend  on  the  cardinality  of  the  generic 

classes  of  the  IS _ A  sons  of  H’,  so  that  an  K  whose  cardinality  is  ten  times  greater  than 

that  of  another  IS _ A  son,  has  a  compatibility  that  is  ten  times  greater.  Also,  any  a 

priori  statistical  information  on  occurrances  of  each  class  could  be  incorporated  here. 
Further  work  along  these  lines  is  left  for  future  research.  A  reasonable  estimate  for 
this  compatibility  for  our  purposes  is  0.4.  If  H’  fails,  then  it  is  highly  unlikely  that  H  will 
succeed  so  H'  removes  support  from  K  with  a  compatibility  of  -1.0. 

4)  This  is  basically  the  inverse  of  type  3.  If  H’  succeeds  then  it  is  completely 
compatible  with  H  (compatibility  =  1.0).  If,  however,  K’  fails,  the  situation  is  the  in¬ 
verse  as  that  in  3)  for  an  H’  that  succeeds.  Therefore,  the  same  discussion  applies. 
However,  the  values  would  be  inverses.  They  could  be  compared  to  the  "frequency  of 
occurrance"  and  "evoking  strengths"  of  the  INTERNIST-11  system  [Pople77]. 

5)  This  compatibility  provides  matching  feedback  for  K  itself,  ,  i.e.,  feedback 
through  the  PART_0F  relationships  in  the  frame.  Failures  in  matching  lead  to  negative 
compatibilities,  while  successes  are  recorded  with  positive  ones.  This  factor  can  be 
tuned  depending  on  noise  characteristics  of  the  images  (high  compatibilities  if  images 
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are  noise  free  and  there  is  high  confidence  in  the  correctness  of  the  kineses  obtained). 
The  values  will  be  further  discussed  in  a  later  section. 

Chapter  7  will  present  guidelines  for  the  determination  of  appropriate  values 
of  the  compatibilities  for  types  1 ,  2,  and  5. 

The  weighting  factor  assigned  to  each  contributing  hypothesis  depends  on  the 
number  of  active  adjacent  hypotheses  and  their  types.  A  total  weight  of  one-half  is 
shared  among  hypotheses  of  types  3,  4,  and  5  (local  evaluation).  The  other  half  is  split 
among  competing  hypotheses  'with  adjacency  types  1  and  2  (global  evaluation).  The 
sum  of  the  weights  must  be  one. 
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6.6  Temporal  Segmentation  and  Instantiation  of  Hypotheses 


Temporal  information  is  a  slot  in  every  frame  and  defines  when  the  motion 
starts  and  when  it  ends.  The  determination  of  the  start  and  end  times  involves  both 
the  choice  of  which  is  the  best  hypothesis  for  an  object  in  a  given  time  interval  and 
which  time  instants  define  the  interval.  We  call  the  determination  of  this  information 

temporal  segmentation. 

As  mentioned  above,  HEARSAY-II  faced  this  same  problem  at  word  boundaries. 
It  used  the  intensity  plot  of  the  utterance  and  depended  on  the  finding  of  a  significant 
local  minimum  in  order  to  determine  word  boundaries  in  time.  The  only  attempt  at 
temporal  segmentation  in  motion  analysis  systems  is  that  of  Badler  [Badler75].  He 
ended  and  started  events  on  changes  of  semantic  components.  For  example,  if  there 
was  a  significant  change  in  a  trajectory  value,  this  would  signal  a  change  of  descriptive 
term.  This  process  entailed  using  very  specific  heuristics  which  required  knowledge  of 
the  contents  of  the  KB.  It  does  not  seem  to  be  a  general  KE  independent  scheme. 

In  our  system,  the  hypotheses  considered  for  temporal  segmentation  are 
those  in  an  object’s  alternative  set  -  the  competitors  due  to  active  similarity  links  and 
those  related  by  the  "next"  constraint. 

Let  us  consider  first  the  simple  case  of  two  hypotheses  competing  in  time, 
neither  of  which  has  any  competitors  outside  the  time  interval  during  which  they  com¬ 
pete.  Intuitively,  we  expect  the  first  hypothesis’  certainty  to  be  1.0  until  the  second 
one  is  activated,  at  which  point  it  abruptly  falls  to  0.5,  At  the  instant  of  activation  they 
have  equal  certainties.  Suppose  that  the  motion  which  corresponds  to  this  second  hy¬ 
pothesis  does  not  begin  for  a  couple  of  images.  During  those  images,  the  certainty  of 
the  first  hypothesis  will  rise,  because  it  is  still  successfully  matching,  and  that  of  the 
second  will  correspondingly  fall.  When  the  second  motion  begins,  the  first  hypothesis 
should  no  longer  match.  Therefore,  its  certainty  will  begin  to  fall,  and  the  certainty  of 
the  second  will  rise.  The  point  at  which  the  second  motion  began  and  the  first  one  end¬ 
ed  is  marked  by  the  maximum  distance  between  the  two  certainty  values  during  the  in- 
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terval  of  competition.  This  defines  their  time  boundary. 

This  maximal  certainty  difference  criterion  is  used  for  segmentation  whenev¬ 
er  the  "next"  constraint  is  present,  regardless  of  its  origin.  However,  complications 
may  arise  when  there  are  several  hypotheses  competing  for  the  same  time  interval, 
i.e.,  related  by  the  similarity  adjacency.  If  a  particular  hypothesis  K  has  several  possi¬ 
ble  "next"  hypotheses,  at  the  instant  H  becomes  a  candidate  for  instantiation,  the  best 
"next"  hypothesis  is  selected  (highest  certainty),  and  the  segmentation  proceeds  as 
above.  If  there  are  several  simultaneous  hypotheses,  the  best  must  be  chosen.  This  is 
done  by  selecting  the  one  -with  the  highest  certainty  at  the  minimum  end  time  of  the 
alternative  set.  If  one  waits  longer,  say  until  the  longest  expected  duration  elapses, 
then  the  already  ended  hypotheses  will  have  their  certainty  values  altered  as  a  result 
of  interactions  with  their  "next"  hypotheses.  Any  "next”  hypotheses  are  then  segment¬ 
ed  with  this  best  one.  Examples  of  the  segmentation  process  are  presented  in  the 
next  chapter. 

When  a  hypothesis  is  selected  as  described  above,  it  is  instantiated.  Instantia¬ 
tion  in  our  system  means  that  a  frame  token  is  created  for  the  hypothesis,  and  that  all 
slots  are  filled  in  that  token.  Note  that  at  this  point,  it  is  not  necessarily  true  that  all 
constraints  on  each  slot  are  true.  The  system  seeks  the  best  match  from  among  many 
hypotheses.  Any  constraint  matching  failures  are  recorded  as  exception  frames  so 
that  it  is  clear  that  the  hypothesis  has  not  totally  succeeded,  and  also  it  is  clear  in 
which  respects  it  has  failed. 


6.7  Contour  Expectation  Generation 

Using  the  current  "best"  motion  hypotheses  for  each  object  in  the  image,  an 
expected  contour  for  it  in  the  next  image  may  be  produced  based  on  the  contour  in  the 
previous  image.  To  start  this  process,  it  is  assumed  that  in  the  first  image  of  the  se¬ 
quence,  a  conceptual  description  is  given  for  each  object  in  the  image.  New  objects 
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are  not  allowed  to  enter  into  the  image.  This  conceptual  description  consists  of  the 
complete  contour  of  each  object  with  all  of  its  parts  labelled,  so  that  the  instantiation 
of  its  corresponding  object  frame  is  a  trivial  matter. 

The  predicted  outline  may  be  determined  in  one  of  two  ways.  The  prediction 
may  simply  be  the  contour  in  the  previous  image  along  with  a  search  window  of  width 
equal  to  twice  the  maximum  allowed  displacement  for  any  point  in  the  problem 
domain.  In  general,  however,  more  information  is  known  about  the  motions. 

If  the  hypothesis  contains  the  expected  motion  parameters,  they  may  be  ap¬ 
plied  to  the  current  contour  to  derive  the  new  one,  and  the  range  of  these  parameters 
will  give  the  search  window.  In  addition,  if  several  hypotheses  are  considered  viable, 
that  is,  multiple  maximal  hypotheses,  then  the  union  of  their  predicted  search  regions 
is  used  as  the  region  while  the  best  contour  is  used  as  guidance  for  edge  orientation. 

Clearly,  predictions  are  only  provided  for  objects  whose  motion  capability  is 
at  least  "mobile’'.  "Fixed"  objects  are  assumed  stationary  forever.  If  a  mobile  object 
does  not  have  an  active  motion  hypothesis  associated  with  it,  (i.e.,  it  has  not  yet 
moved),  the  expectation  for  it  is  derived  by  the  simple  method  described  above. 
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CHAPTER  7 


EXAMPLES 


7.1  The  System  -  ALVEN 

The  system  we  have  partially  implemented  is  called  ALVEN  -  A  Left  Ventricu¬ 
lar  Wall  Motion  Analysis  System.  To  this  point,  frames  have  been  defined  for  normal, 
central  apex  ischemia,  central  posterior  segment  ischemia  and  central  anterior  seg¬ 
ment  ischemia.  This  entails  73  LV  motion  frames,  26  general  motion  frames,  11  gen¬ 
eral  object  frames,  and  7  LV  object  frames.  The  current  level  of  detail  of  description 
specifies  how  segments  move  and  in  what  directions.  However,  cardiologists  do  not 
know  the  expected  velocities  nor  exact  trajectories  for  each  phase  of  the  cycle.  The 
data  collection  system  (to  be  described)  is  designed  to  augment  our  knowledge  in 
these  matters. 

Several  portions  of  the  system  were  implemented,  namely  the  KB  mainte¬ 
nance  system,  the  image  analysis  component  (including  both  the  essential  trace  and 
kineses  determinations),  the  hypothesis-kineses  matching  process,  and  the  focus  of  at¬ 
tention  mechanism.  We  assume  that  in  testing  such  a  large  system,  if  the  parts  work, 
then  the  whole  does  as  well.  Each  component  was  therefore  tested  independently. 
Descriptions  of  the  components  along  with  examples  and  results  are  presented  in  the 
remainder  of  the  chapter. 

The  ALVEN  project  has  already  spawned  several  other  projects  -  a  knowledge 
acquisition  and  verification  system,  a  diagnosis  system  and  a  knowledge  integration 
system. 

1  gratefully  acknowledge  programming  assistance  from  Steve  Hume,  Bill 
Reeves,  Steve  Ko-Tai  and  especially  Jim  Delgrande. 


-  122  - 


7.2  The  Knowledge  Base  Compiler 

A  syntax  for  the  representation  formalism  was  defined  and  a  parser  con¬ 
structed  for  it  [Norman78].  Parser  tokens  are  then  translated  into  the  internal  struc¬ 
ture  of  the  KB  -  using  programming  language  structure  definitions  and  an  elaborate 
symbol  table.  Constraints  were  coded  as  functions  whose  parameters  are  the  variables 
used  within  the  constraint.  This  module  not  only  allows  one  to  enter  an  entire 
knowledge  base,  but  also  allows  interactive  editing  and  modifications.  That  is,  the  user 
may  enter  frames  in  pieces  and  alter  them  in  that  way  as  well,  at  either  the  frame  or 
slot  level.  Syntax  checking  is  done  regardless  of  the  way  in  which  the  frames  or  slots 
are  added  or  deleted.  Path  access  names  are  constructed  for  variables  in  order  to  fa¬ 
cilitate  accesses  to  them  during  processing. 

7.3  The  Image  Analysis  Module 

Examples  of  the  prediction  guided  edge  detection  process  were  presented 

earlier.  Let  us  consider  the  remaining  portions  of  this  module.  Consider  the  arbitrary 

shape  in  Fig.  7.1.  The  essential  trace  for  describing  this  shape  is: 

token  shape_l  instance_of  C0NTRACT1LE_0BJECT  with 
centroid  :  point_l; 
long_axis  :  str_line_l  ; 
lat_axis  :  str_line_2  ; 
area  :  117; 
arclen  :  42  ; 
seg_l  :  curve_l  ; 
seg_2  :  curve_2  ; 
seg_3  :  curve_3  ; 
seg_4  :  curve_4  ; 
seg_5  :  curve_5  ; 
seg_6  :  curve_6  ; 
seg_7  :  curve_7  ; 

end 

This  token  defines  the  characteristics  of  the  shape  as  a  whole  as  wTell  as  the  fact  that  it 

is  made  up  of  seven  shape  features. 

token  point_l  instance_of  POINT  with 
x  :  8.5  ; 
y  :  16.5  ; 

end 

token  str_line_l  instance_of  STR_LINE  with 
start_pt  :  point_2  ; 
end _ pt  :  point_3  ; 
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arclen  :  15  ; 


end 

token  point_2  instance_of  POINT  with 
x  :  8  ; 
y  :  23.5  ; 

end 

token  point„3  instance__of  POINT  with 
x  :  9  ; 
y  :  8.5  ; 

end 

token  str__line__2  instance__of  STFL-LINE  with 
start_pt  :  point_4  ; 

end _ pt  :  point_5  ; 

arclen  :  14  ; 

end 

token  point_4  instance_of  POINT  with 
x  :  1  ; 
y  :  16  ; 

end 

token  point_5  instance_of  POINT  with 
x  :  14.5  ; 
y :  17  ; 

end 

token  curve_l  instance_of  CURVE_SEG  with 
start_pt  :  point__6  ; 

end _ pt  :  point_7  ; 

depth_pt  :  point_8  ; 

mouth _ len  :  6.5  ; 

depth  :  3.5  ; 
arclen  :  9.6  ; 
shape  :  bay  ; 

end 


There  is  no  specific  purpose  served  by  presenting  the  remainder  of  the  tokens 
for  all  the  points  and  curved  segments  of  the  shape  in  the  figure.  The  remainder  of  the 
curves  are  of  types:  shpoint  (short  point),  lallagoon  (large  left  lagoon),  cape,  smrlagoon 
(small  right  lagoon),  lopoint  (long  point),  bay,  and  lopoint  respectively  traversing  coun¬ 
terclockwise  from  curve_l.  The  form  and  details  of  the  essential  trace  should  be  clear 
from  the  above. 

The  left  ventricle  is  not  described  currently  in  terms  of  its  shape  features. 
The  reason  for  this  is  that  nothing  is  known  about  what  the  ’’proper”  shape  is  for  the 
LV  (there  may  not  even  be  "proper”  normal  or  abnormal  shapes).  Therefore,  as  men- 


-  124  - 


tioned  in  chapter  5,  we  use  the  characterization  found  in  the  literature  of  three  seg¬ 
ments  and  a  centroid.  In  that  chapter  as  well,  the  frames  for  the  LV  as  an  object  were 
shown.  The  essential  trace  of  the  LV  would  consist  of  tokens  with  that  form.  In  addi¬ 
tion,  kineses  are  calculated  for  those  segments  and  for  the  LV  as  a  whole.  Figure  7.2 
shows  the  six  consecutive  outlines  on  which  we  tested  our  program  for  calculation  of 
the  essential  kineses.  The  essential  kineses  for  the  first  two  of  these  consecutive  out¬ 
lines  shown  in  Fig  7.2  are: 


token  point_  1  instance_of  POINT  with 
x :  143  ; 

y  :  89  ; 

end 

token  point_2  instance_of  POINT  with 
x  :  143  ; 

y  :  89  ; 

end 

token  point_3  instance_of  POINT  with 
x :  151  ; 

y  :  31  ; 

end 

token  point_4  instance_of  POINT  with 
x :  157  ; 

y  :  35  ; 

end 

token  point_5  instance_of  POINT  with 
x  :  44  ; 

y :  175  ; 

end 

token  point_6  instance_of  POINT  with 
x  :  47  ; 

y :  175  ; 

end 

token  point_7  instance_of  POINT  with 
x  :  75  ; 

y ;  205  ; 

end 

token  point_8  instance_of  POINT  with 
x  :  78  ; 

y :  202  ; 

end 

token  point_9  instance_of  POINT  with 
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x :  134  ; 

y:  181; 

end 

token  point_10  instance_of  POINT  with 
x :  134  ; 

y:  181; 

end 


token  point_ll  instance_of  POINT  with 
x  :  121  ; 

y  :  50  ; 

end 


token  point_12  instance_of  POINT  with 
x  :  121  ; 

y  :  44  ; 

end 


token  point_13  instance_of  POINT  with 
x :  173  ; 

y :  100  ; 

end 

token  point_14  instance_of  POINT  with 
x  :  168  ; 

y  :  85  ; 

end 


token  point_15  instance_of  POINT  with 
x  :  131  ; 

y :  185  ; 

end 

token  point_16  instance_of  POINT  with 
x  :  131  ; 

y :  1B9  ; 

end 

token  antseg  instance_of  CURVED_SEG  with 
start_pt  :  point_5  ; 

end _ pt  :  point_ll  ; 

shape  :  curve  ; 
depth_pt  :  ?  ; 
mouth_len  :  146  ; 
depth  :  ?  ; 
area  :  5861 ; 

mot_cap  :  contractile  ; 
arclen  :  167  ; 

end 

token  apicseg  instance_of  CURVED_SEG  with 
start_pt  :  point_ll  ; 

end _ pt  :  point_13  ; 

shape  ;  curve  ; 
depth_pt  :  ?  ; 
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mouth_len  :  71  ; 
depth  :  ?  ; 

mot__cap  :  contractile  ; 
arclen  :  1 12  ; 

end 

token  postseg  instance_of  CURVED_SEG  with 
start_pt  :  point_13  ; 

end _ pt  :  point_9  ; 

mot_cap  :  contractile  ; 
shape  :  curve  ; 
depth_pt :  ?  ; 
mouth_len  :  90  ; 
depth  :  ?  ; 
arclen  :  96  ; 

end 

token  lv  instance_of  CL0SED_0UTLINE  with 
mot_cap  :  contractile  ; 
centroid  :  point_l  ; 
lat_axis  :  lataxis  ; 
long_axis  :  longaxis  ; 

apical _ seg  :  apicseg  ; 

anterior_seg  :  antseg  ; 
posterior_seg  :  postseg  ; 
area  :  14470  ; 
arclen  :  375  ; 
shape  :  ?  ; 

end 

token  mitv  instance_of  LINE_SEG  with 
start_pt  :  point_7  ; 
end_pt  :  point_5  ; 
mot_cap  :  mobile  ; 
arclen  :  43  ; 

end 

token  aortv  instance_of  OBJECT  with 
start_pt  :  point_9  ; 
end_pt  :  point_7  ; 
mot_cap  :  mobile  ; 
arclen  :  63  ; 

end 

token  longaxis  instance_of  STR_LINE  with 
start_pt  :  point_3  ; 
end_pt  :  point_9  ; 
mot_cap  :  contractile  ; 
arclen  :  150  ; 

end 

token  lataxis  instance_of  STR_LINE  with 
start_pt  :  point_5  ; 
end_pt  :  point_15  ; 
mot_cap  :  contractile  ; 
arclen  :  87  ; 
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end 


token  time_interva] _ 1  instance_of  TIME_INTERVAL  with 

start_time  :  0  ; 

end time  :  1  ; 

duration  :  1  ; 

end 

token  location _ change_l  instance_of  L0CAT10N_CHANGE  with 

subj  :  point_l  ; 

time_int  :  time_intervaJ _ 1  ; 

start_loc  :  point_l  ; 
end _ loc  :  point_2  ; 

end 

token  location _ change_2  instance_of  LOCATION_ CHANGE  with 

subj  :  point_3  ; 

time_int  :  time_interval _ 1  ; 

start_loc  :  point_3  ; 
end _ loc  :  point_4  ; 

end 

token  location _ change_3  instance_of  L0CAT10N_CHANGE  with 

subj  :  point_5  ; 

time_int  :  time_interval _ 1  ; 

start_loc  :  point_5  ; 
end _ loc  :  point_6  ; 

token  location _ change_4  instance_of  LOCAT]ON_CHANGE  with 

subj  :  point_7  ; 

time_int  :  time_interval _ 1  ; 

start_loc  :  point_7  ; 
end _ loc  :  point_S  ; 

end 

token  location _ change_5  instance_of  L0CAT10N_CHANGE  with 

subj  :  point_9  ; 

time_int  :  time_interva] _ 1  ; 

start_loc  :  point_9  ; 
end _ loc  :  point_10  ; 

end 

token  location _ change_6  instance_of  L0CAT10N_CHANGE  with 

subj  :  point_ll  ; 

time_int  :  time_interval _ 1  ; 

start_loc  :  point_ll  ; 
end _ loc  :  point_12  ; 

end 

token  location_change_7  instance_of  L0CAT10N_CHANGE  -with 
subj  :  point_13  ; 

time_int  :  time_interval _ 1  ; 

start_loc  :  point_13  ; 
end_loc  :  point_14  ; 

end 
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token  location _ change_8  instance__of  L0CAT10N_CHANGE  with 

subj  :  point_15  ; 
time_int  :  time_intervaL_l  ; 
start__loc  :  point„15  ; 
end _ loc  :  point_16  ; 

end 

token  length_change_l  instance_of  LENGTH_CHANGE  with 
subj  :  antseg  ; 

time__int  :  time_interval _ 1  ; 

start_l :  167  ; 

end _ 1  :  174  ; 

end 

token  length _ change_2  instance_of  LENGTH_CHANGE  with 

subj  :  apicseg  ; 

time_int  :  time_interval _ 1  ; 

start_l  :  112; 

end _ 1  :  92  ; 

end 

token  length_change_3  instance_of  LENGTH_CHANGE  -with 
subj  :  postseg  ; 

time_int  ;  time_interval _ 1  ; 

start_l  :  96  ; 

end_l  :  177 ; 

end 

token  length _ change_4  instance_of  LENGTH_CKANGE  with 

subj  :  lv  ; 

time_int  :  time_intervaJ _ 1  ; 

start_l :  375  ; 

end _ 1  :  443 ; 

end 

token  length _ change_5  instance_of  LENGTK_CHANGE  with 

subj  :  mitv  ; 

time_int  ;•  time_interval_l  ; 
start_l  :  43  ; 

end _ 1  :  41  ; 

end 

token  length_change_6  instance_of  LENGTH_CHANGE  with 
subj  :  aortv  ; 

time_int  :  time_interval _ 1  ; 

start_l  :  63  ; 

end_l  :  59 ; 

end 

token  length _ change_7  instance_of  LENGTH_CHANGE  with 

subj  :  long  axis  ; 

time_int  :  time_interval_l  ; 

start_l :  150  ; 

end _ 1  :  147  ; 

end 
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token  length_change_8  Lnstance_of  LENGTH_CHANGE  with 
subj  :  lataxis  ; 

time_int  :  time_interval_l  ; 
start_l  :  87  ; 

end_l  :  85  ; 

end 

token  area__change_l  instance_of  AREA _ CHANGE  with 

subj  :  antseg  ; 

time_int  :  time_interval _ 1  ; 

start_a  :  5861  ; 

end _ a  :  5943  ; 

end 

token  are  a__c  hang  e_2  instance_of  AREA _ CHANGE  with 

subj  :  apicseg  ; 

time_int  :  time_intervaJ _ 1  ; 

start_a  :  2187  ; 

end_a  :  1840  ; 

end 

token  area_change_3  instance_of  AREA _ CHANGE  with 

subj  :  postseg  ; 

time_int  *  time_interva] _ 1  ; 

start_a  :  2255  ; 

end_a  :  4098  ; 

end 

token  area_change_4  instance_of  AREA-CHANGE  with 
subj  :  lv  ; 

time_int  :  time_Lnterva] _ 1  ; 

start_a  :  14470  ; 
end _ a  :  15910  ; 

end 

Note  that  values  for  the  slots  "depth”  and  "depth_pt"  are  not  currently  calculated. 

7.4  The  Hypothesis  Matching  Process 

Matching  proceeds  as  described  in  chapter  6,  taking  as  input  the  essential 
kineses.  Levels  of  description  are  matched  and  any  re-computations  are  done.  For 
standard  types  of  constraints,  function  calls  determine  whether  or  not  matching 
succeeds.  However,  when  the  "for"  constraint  is  used,  a  search  of  possible  candidate 
tokens  determines  the  slot  instance.  The  first  token  that  satisfies  the  constraints  is 
used.  Complete  constraints  are  always  checked,  i.e,  the  inherited  ones  as  well  as  the 
local  ones,  so  there  is  no  need  for  backtracking. 

The  slot  constraints  are  set  up  so  that  matching  is  facilitated.  Each  constraint 
group  for  a  slot  is  implemented  as  a  parameterized  function  -  one  such  function  per 
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slot.  Data  structures  are  set  up  so  that  any  variable  can  be  accessed  through  its  path 
name,  i.e.,  each  variable  is  prefixed  by  the  name  of  the  frame  it  is  defined  in,  and  then 
this  new  construct  is  treated  as  a  variable  so  the  process  is  recursive.  It  applies  for 
generic  frames  as  well  as  for  tokens.  In  addition,  the  access  mechanism  uses  the  IS_A 
hierarchy  for  inherited  slot  names. 

When  the  constraint  function  returns  false  or  a  search  does  not  discover  an 
appropriate  instance,  an  exception  token  is  created  and  added  to  the  instance  KB.  The 
exception  token  is  either  the  one  specified  in  the  slot  with  the  failing  constraint,  or  one 
of  the  system  exception  types  depending  on  the  slot  name. 

7.5  Hypothesis  Certainty  Factors  and  Temporal  Segmentation 

Several  examples  of  the  time  course  of  certainties  obtained  via  conceptual 
adjacency  and  the  relaxation  process  will  now  be  presented.  They  will  be  given  in 
terms  of  graphs  of  certainty  vs.  time  instant  for  each  hypothesis  in  consideration.  The 
values  were  obtained  from  our  implentation  of  the  focus  of  attention  program.  In  addi¬ 
tion,  the  matching  progress  for  each  hypothesis  is  shown  on  the  graph  as  well  as  the 
conceptual  adjacency  relations. 

Several  relationships  between  system  response  time  and  compatibility  factors 
arise  from  these  examples.  They  will  be  discussed  as  they  appear. 

7.5. 1  Competition  Among  Similar  Hypotheses 

Two  types  of  similarity  competition  will  be  presented.  The  first,  in  Fig.  7.3, 
shows  only  two  similar  hypotheses  over  a  common  time  interval  and  demonstrates  how 
consistent  matching  history  leads  to  the  selection  of  the  correct  hypothesis.  The 
second,  in  Fig.  7.4,  shows  several  hypotheses  in  competition,  one  of  them  being  the 
correct  one,  but  with  random  noise  added  to  the  matching  process.  This  demonstrates 
how  the  inertia  of  the  feedback  system  attenuates  the  effects  of  isolated  noise  stimuli. 

Two  basic  properties  of  the  control  structure  can  be  seen  in  those  two  figures 
a  time  constant  and  a  noise  sensitivity.  To  consider  the  time  constant  first,  note  that 
the  curves  in  Fig.  7.3  look  very  much  like  exponential  functions.  In  electronics,  when 
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the  response  of  a  circuit  exhibits  such  a  shape,  a  useful  parameter  which  is  calculated 
is  the  system  "time  constant".  In  particular,  if  the  curve  can  be  approximated  by  the 
function: 

V  =  V0(l-e  T  ) 

for  the  rising  curve,  where  V  is  the  system  output  and  VD  is  the  final,  stable  system 
output,  then  t  is  called  the  "time  constant”  of  the  system.  In  other  words,  the  output 
comes  to  within  1  /  e  of  its  final  value  in  a  time  interval  equal  to  t.  We  can  determine 
values  for  t  in  our  system  as  well,  with  an  analogous  interpretation.  If  we  use  the 
definition  of  the  critical  value  given  above,  we  can  say  that  the  system  must  analyze  t 
images  in  order  to  discriminate  between  two  competing  hypotheses  under  ideal  match¬ 
ing  conditions.  In  the  example  given,  this  response  time  is  2.  However,  we  must  be 
aware  of  the  fact  that  the  similarity  competition  is  not  the  only  rating  mechanism 
operating.  There  is  also  the  PART_0F  adjacency  to  take  into  consideration.  Therefore, 
let  us  perform  the  test  described  in  the  the  preceeding  paragraphs  and  illustrated  in 
Fig.  7.3,  for  all  combinations  (at  0.1  intervals,  that  is)  of  similarity  and  PART_0F  com¬ 
patibility.  For  each  test,  there  are  one  hundred  of  them,  we  will  determine  the  r,  thus 
plotting  a  family  of  curves  of  PART_0F  compatibility  on  the  axes  t  versus  similarity 
compatibility.  This  is  shown  in  Fig.  7.5.  From  this  graph,  we  can  see  the  relationship 
between  the  two  compatibility  types  and  their  associated  response  times.  Note  that 
the  system  can  do  no  better  than  a  response  time  of  3  for  our  control  structure.  We 
can  also  see  which  combinations  of  values  lead  to  reasonable  values  and  which  lead  to 
very  poor  response  times. 

The  PART_0F  compatibility  can  be  viewed  as  a  "noise  sensitivity"  parameter. 
The  greater  the  probability  of  erroneous  essential  kineses,  then  the  smaller  the 
PART_0F  compatibility  should  be.  Fig.  7.4  showed  several  competing  hypotheses  with  a 
similarity  compatibility  of  1.0  and  a  PART_0F  compatibility  of  0.8.  If  the  latter  value  is 
changed  to  0.6,  the  curves  in  Fig.  7.6  are  seen.  It  is  clear  that  the  leading  hypothesis 
exhibits  much  less  amplitude  change  on  matching  failures  than  in  Fig.  7.4,  and  it  still  is 
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the  winner  in  the  end.  However,  the  price  that  is  paid  is  that  the  final  values  are  not 
very  far  apart.  The  differences  are  not  as  significant  as  in  the  previous  case.  That  is, 
there  is  a  low  "selectivity"  among  the  hypotheses.  Also  note  that  the  final  order  of  the 
hypotheses  is  the  same. 

The  central  point  of  these  examples  is  that  they  suggest  an  empirical  method 
for  determining  the  compatibilities  needed  for  the  updating  function.  Noise  effects  can 
be  analyzed  and  rated  on  a  scale  of  0.0  to  1.0.  0.0  corresponds  to  very  noisy  imagery. 
This  rating  becomes  the  compatibility  for  the  adjacencies  of  type  5.  Depending  on 
this  rating,  the  curves  are  used  to  select  the  best  values  of  type  1  compatibility  for 
that  particular  type  5  compatibility.  The  final  constraint  is  that  the  event  of  shortest 
possible  duration  in  the  domain  must  be  recognizable.  This  determines  the  r  value  for 
the  sj^stem.  Using  the  r  and  the  compatibility  value  for  type  5  adjacencies,  we  can  find 
the  set  of  reasonable  values  for  the  similarity  compatibility.  If  several  values  are  appli¬ 
cable,  the  one  with  the  fastest  response  time  may  be  preferred. 

Let  us  further  examine  the  relationships  between  the  similarity  and  PART_0F 
compatibilities.  The  above  analysis  takes  only  two  simultaneous  hypotheses  into  ac¬ 
count.  In  reality,  there  may  be  several  hypotheses  competing  with  one  another.  In  ad¬ 
dition,  noise  is  not  related  to  the  PART_0F  compatibility  in  the  preceeding  discussion. 

If  noise  is  present  in  the  images  then  it  is  reasonable  to  assume  that  some  of 
the  essential  kineses  that  are  determined  will  incorrectly  reflect  the  images.  There¬ 
fore,  the  proper  hypotheses  will  fail  in  the  matching  with  them  wioile  erroneous  hy¬ 
potheses  will  succeed.  For  this  reason,  in  the  presence  of  noise,  we  do  not  wish  the 
certainty  factor  changes  to  be  as  great  as  in  the  case  of  noisefree  images.  To  account 
for  this,  the  PART_0F  compatibility  can  be  set  at  values  that  reflect  the  noise  in  the 
images.  Let  us  define  10%  noise  to  be  present  in  the  data  if  one  out  of  ten  matches  of 
hypothesis  to  kinesis  fails  w'hen  it  should  have  matched.  Similarly,  20%  noise  if  twTo  out 
of  ten  fail,  etc.  The  noise  percentage  determines  the  amount  of  decrease  in  PART_0F 
compatibility  from  the  noisefree  case.  Therefore,  in  10%  noise,  the  PART_0F  compati- 
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bility  is  0.9,  for  20%  0.8,  etc. 

The  graph  in  Figure  7.7  shows  the  results  of  tests  of  the  certainty  updating 
mechanism  under  the  following  conditions: 

•  The  trials  are  run  using  a  sequence  of  random  numbers  to  represent  matching 
successes  and  failures,  and  are  adjusted  for  noise  levels.  For  example,  if  there  is  a  10% 
noise  level,  then  any  random  number  below  0.1  is  considered  to  be  a  matching  failure 
for  the  correct  hypothesis,  while  the  same  range  is  used  for  matching  successes  for  the 
incorrect  ones.  The  range  of  random  numbers  is  0.0  to  1.0. 

•  Each  point  in  the  figure  is  the  average  time  value  obtained  over  thirty  trials  at 
the  same  values  for  similarity  and  PART_0F  compatibility  and  number  of  competitors. 

•  The  decision  threshold,  that  is,  the  certainty  value  that  must  be  achieved  before 
a  particular  hypothesis  is  instantiated  is  related  to  the  number  of  hypotheses  in  the 
following  manner.  The  t*  is  the  time  at  which  the  certainty  of  a  hypothesis  reaches 

which  simplifies  to: 

0.632+  °-'36-9 

Ni 

This  has  been  determined  using  the  assumption  that  each  of  the  hypotheses  that  are 
competing  start  with  equal  certainty  value  which  is  determined  using  the  number  of 
hypotheses  Ni 

•  Similarity  compatility  is  set  to  1.0  in  all  cases.  The  PART_0F  compatibility 
reflects  the  noise  level  present.  For  example,  if  the  noise  level  is  10%,  then  the 
PART_0F  compatibility  is  set  to  0.9  in  order  to  reflect  the  fact  that  there  is  a  degree  of 
uncertainty  in  the  data  and  that  changes  in  certainty  value  should  not  be  as  great  as 
when  the  data  is  completely  certain. 

•  Noisefree  and  10%,  20%,  30%,  40%  and  50%  noise  images  were  tested. 

•  Each  test  was  run  on  from  two  to  sixteen  simultaneous  competitors. 

The  results  are  approximated  by  linear  relationships,  although  there  is  no 
reason  why  we  should  believe  that  the  relationships  should  be  linear.  Further 


-  134  - 


134a 


mathemtical  analysis  of  the  updating  rule  is  required  for  that.  The  50%  noise  line  is 
very  close  to  vertical  -  thus  confirming  the  expectation  we  get  from  information  theory 
that  at  50%  noise  there  is  no  significant  data  in  the  images.  The  remainder  of  the  lines 
verify  our  intuition  in  expecting  that  the  noisier  the  data,  the  more  of  it  that  is  re¬ 
quired  in  order  to  make  decisions  with  the  same  confidence. 

Another  interesting  observation  can  be  made  from  this  graph.  If  there  are, 
for  example,  5  competing  hypotheses,  under  10%  noise  conditions,  then  the  system 
must  examine  at  least  10  inter-image  descriptions  (11  images)  in  order  to  discriminate 
from  among  them.  This  means  that  if  the  motion  concepts  represented  by  the  hy¬ 
potheses  have  expected  minimum  duration  D,  then  the  sampling  rate  of  the  film  must 
be  at  least  ll/D  in  images  per  second.  This  can  be  generalized  to  the  following  rela¬ 
tionship: 


Sfi  niin- 


max  1) 

i  duTi 


where  SRrair  is  the  minimum  required  sampling  rate,  and  t*  is  the  minimum  number 
of  inter-image  descriptions  required  determined  from  the  graph,  for  discriminating 
from  among  a  set  of  hypotheses  \¥i],  cardinality  of  which  is  AT*  and  whose  minimum 
duration  is  duri. 

This  relationship  has  important  implications.  It  can  be  used  to  determine 
whether  or  not  a  particular  image  sequence  related  to  a  specific  knowledge  base,  can 
be  analyzed  using  the  methods  presented  in  this  document. 

7.5.3  Competition  Between  "Next"  Hypotheses 

Fig.  7.8  shows  two  hypotheses  in  competition  during  a  short  time  interval 
under  correct  "noiseless”  matching.  Note  that  the  second  begins  at  time  instant  6.  In 
this  figure,  the  certainty  (vertical)  axis  is  distorted  in  order  to  show  the  peak  in  cer¬ 
tainty  value  at  which  the  segmentation  occurs  using  the  rules  discussed  in  Chapter  6. 
The  point  of  maximum  separation  of  the  certainty  factors  for  the  two  hypotheses  dur¬ 
ing  the  time  interval  of  competition  is  their  temporal  boundary. 

An  example  of  a  longer  overlap  period  is  shown  in  Fig.  7.9.  This  figure  is  in- 
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tended  to  illustrate  the  amount  of  time  required  for  hypothesis  certainty  overtake  dur¬ 
ing  a  "next”  competition  interval.  Therefore,  the  certainty  axis  is  not  distorted  as  in 
the  previous  figure  -  however,  the  same  characteristic  hump  in  the  curve  is  present  in 
the  certainty  factors,  although  it  is  not  apparent  from  the  figure.  We  see  here  that  the 
"next"  hypothesis  overtakes  the  first  one  after  8  time  instants  from  the  point  at  which 
the  motion  it  represents  actually  begins.  Therefore,  if  no  mechanism  is  provided  to 
determine  where  the  true  start  time  is,  i.e.,  temporal  segmentation  as  described 
above,  the  system  would  conclude  that  the  motion  started  8  time  instants  later  than  it 
really  did. 

Again,  the  PART_OF  compatibility  interacts  with  the  "next"  compatibility.  In 
the  figure,  these  values  were  0.8  and  1.0  respectively.  Let  us  plot  the  time  instant  when 
the  "next"  motion  hypothesis  overtakes  the  first  one,  with  regard  to  certainty  factor, 
for  all  combinations  of  compatibility  values  for  the  two  interacting  mechanisms.  This  is 
shown  in  Fig.  7.10.  From  this  we  see  that  no  combination  leads  to  a  faster  overtake 
than  in  6  images.  Therefore,  a  temporal  segmentation  procedure  is  not  needed  if  the 
time  interval  represented  by  6  images  is  insignificant  for  the  error  factor  in  the  start 
and  end  times  of  events.  In  LV  motion,  this  is  a  significant  time  interval  and  therefore 
the  segmentation  is  necessary.  Note  also  from  the  figure  that  computationally  reason¬ 
able  time  intervals  occur  only  if  the  two  compatibility  values  are  set  quite  high. 

The  examples  just  presented  answer  the  questions  we  raised  during  discus¬ 
sion  of  the  inertia  based  focus  mechanism  in  Chapter  6.  The  mechanism  itself  is  in¬ 
dependent  of  problem  domain  and  knowledge  frame  contents.  However,  the  domain  is 
taken  into  account  in  the  compatibility  values.  Although  much  further  experimenta¬ 
tion  is  necessary,  these  results  support  our  belief  that  an  inertia  based  focus  mechan¬ 
ism  is  indeed  satisfactory  for  motion  understanding. 

7.6  The  Prediction  Generator 

The  prediction  generator  uses  the  current  essential  trace  for  each  object  and 
the  current  best  hypotheses  for  each  object’s  motion.  A  function  is  required  which 
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for  all  values  of  PART_OF  compatibility 


given  a  hypothesis  and  a  time  instant,  returns  the  portions  of  it,  as  a  frame  token,  that 
apply  for  that  particular  time  instant  including  both  the  default  values  and  the  ranges. 
Given  these  two  inputs,  it  is  a  simple  matter  to  calculate  a  prediction.  However,  it 
should  be  clear  that  such  a  scheme  is  not  possible  unless  the  proper  data  is  stored  in 
the  KB.  There  needs  to  be  a  default  for  the  cases  where  exact  velocities  are  not  known. 
This  default  is  domain  dependent  and  may  be  of  the  form  "no  object  moves  more  than 
X  cm.  during  any  inter-image  interval". 

However,  we  do  have  some  situations  in  which  velocities  or  at  least,  relative 
velocities  are  known.  In  most  of  the  LV  motion  frames,  during  any  one  phase,  the  velo¬ 
cities  of  each  segment  are  compared  to  those  of  the  segment’s  spatial  neighbours.  In 
normal  cases,  the  motion  patterns  are  symmetrical  -  velocities  of  each  segment  are 
approximately  the  same  in  magnitude  during  each  phase.  In  addition,  the  general  tra¬ 
jectory  is  known  (TOWARDS  or  A\VAY_FROM  the  centroid).  In  this  case,  the  essential 
trace  for  each  segment  is  moved  in  the  proper  direction  by  the  maximum  distance  that 
it  can  travel  in  one  interval.  The  resulting  contour  becomes  one  of  the  limits  of  the 
prediction  window.  If  the  segment  may  have  some  motion  in  the  opposite  direction 
(during  isovolumic  relaxation  for  example),  the  segment  is  moved  to  that  limiting  posi¬ 
tion  in  order  to  form  the  other  limit  of  the  window.  If  it  cannot,  then  the  limit  is  the 
current  location  of  the  segment.  The  orientation  of  the  segment  is  set  to  be  the  same 
as  that  of  the  current  one.  This  is  mainly  because  no  LV  shape  information  is  available 
in  the  KB.  If  we  knew  the  shape  changes  of  each  segment  between  phases,  then  the 
prediction  could  include  a  more  informed  scheme  for  orientation  prediction. 

For  example,  if  the  current  best  hypothesis  were  1SCH_CAP_LV__ CYCLE,  and 
the  apical  segment  were  being  considered  during  the  isovolumic  relaxation  phase,  the 
prediction  could  be  quite  precise  as  to  location.  This  is  because  we  know  that  during 
this  phase,  the  apical  segment  may  move  up  to  2  mm  AWAY_FROM  the  centroid  or  up 
to  6  mm.  TOWARDS  the  centroid.  Given  the  current  segment  location,  a  search  win¬ 
dow  of  width  8  mm.  is  created  around  the  segment,  with  the  predicted  orientation  be- 
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ing  the  same  as  that  in  the  current  segment. 

7.7  The  Data  Collection  System 

This  system  [Delgrande79]  has  two  components:  the  acquisition  component 
and  the  analysis  component  .  The  data  acquisition  program  allows  a  user,  given  a  set  of 
digitized  cardiac  images,  to  specify  the  border  of  the  LV  along  with  other  points  and 
features  of  interest.  These  features  may  also  be  grouped  and  named.  For  example,  the 
user  may  outline  the  mitral  and  aortic  valves,  the  LV  wall,  the  papillary  muscles,  the 
diaphragm,  etc.  The  program  also  extracts  the  segments  of  the  LV  -wall  according  to 
the  definitions  discussed  in  chapter  5,  and  computes  the  centroid  and  axes.  A  directo¬ 
ry  of  patients  classified  by  disease  and  LV  size  is  maintained  for  the  data. 

The  analysis  portion  of  the  system  derives  the  essential  trace  for  each  image 
and  the  essential  kineses  for  each  pair  of  consecutive  images.  A  simplified  version  of 
the  shape  matching  procedure  outlined  in  chapter  6  was  implemented;  however,  the 
main  thrust  of  data  collection  at  the  present  is  to  determine  the  motions  of  the  three 
segments  of  the  LV  wall.  The  kineses  for  each  patient  are  stored  in  the  file  system. 
Once  a  substantial  number  of  patients  have  been  analyzed,  average  velocities  and  tra¬ 
jectories  can  be  found  for  each  disease  and  LV  size.  This  information,  along  with  the 
maximum  and  minimum  values  can  then  be  incorporated  into  the  knowledge  base. 

Fig.  7.11  shows  several  screen  images  of  various  phases  of  the  data  collection 
process.  Fig.  7.11a  shows  an  outline  with  the  tracing  menu,  Fig.  7.11b  shows  consecu¬ 
tive  superimposed  images,  Fig.  7.11c  show's  the  apical  segment  traced  over  two  time 
instants  from  which  kineses  are  derived  and  Fig.  7 lid  shows  the  LV  outline  with  the 
aorta  removed  and  the  longitudinal  axis  and  centroid  drawn  in  for  two  consecutive  im¬ 
ages.  This  is  the  outline  for  which  area  and  arclength  are  determined. 
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FIGURE  7.11a 

A  LV  Outline  and  System  Menu 


ml  sc  .  1 
m  i  s  c  .  2 
misc.3 


138a 


f  rame :  1  all 

lv 

misc.l 

apical  see 

6  tep  :  1  >  l  ab 

l  paplLary 

misc  .2 

anterior  teg 

rets  in :  1  der 

r  pap  i  la  ry 

m  i  s  c  .  3 

posterior  seg 

list:  0  2 

aorta 

aort  ic  valve 

next 

ribs 

length  axis 

label 

d  iaphra  gm 

centro id 

intens 

catheter 

snap 

mitral  valve 

f  i  Lm 

apex  pt 

ex  i  t 

po  int  s 

138b 


FIGURE  7.11c 

Apical  Segment  Motion 
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FIGURE  7. lid 

Two  Outlines,  Longitudinal  axis, 
Centroid,  and  Apex  Labelled 
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CHAPTER  8 


CONCLUDING  REMARKS 


A  framework  for  the  abstraction  of  motion  concepts  from  a  sequence  of  im¬ 
ages  has  been  presented.  The  methodology  can  recognize  motions  that  occur  in  an  im¬ 
age  sequence  and  can  classify  them  according  to  the  motion  types  stored  in  a 
knowledge  base.  The  knowledge  representation  formalism  has  several  characteristics: 

•  motion  concepts  are  represented  using  frames,  with  slots,  constraints  and  de¬ 
fault  values; 

•  the  frames  are  organized  into  an  IS _ A  hierarchy,  in  order  to  relate  general  and 

specific  motion  concepts; 

•  an  implicit  PART_OF  hierarchy  is  provided  in  that  slots  are  PART_OF  the  frame 
they  belong  to,  allowing  for  varying  levels  of  abstraction  of  description; 

•  similarity  links  are  included  in  the  frames,  whose  main  feature  is  the  difference 
expression  that  describes  the  time  sequence  of  differences  between  two  related  mo¬ 
tions,  such  that  once  observed  in  the  image  sequence,  these  differences  aid  in  the 
selection  of  alternate  motion  hypotheses; 

•  temporal  data  is  explicitly  represented  as  a  start  time,  end  time  and  duration 
for  each  motion.  Any  inter-event  timing  relationships  can  be  expressed  in  terms  of 
boolean  relations  of  arithmetic  predicates  made  up  of  these  data.  These  include  "be¬ 
fore",  "after”,  "same  time"  and  other  more  complex  descriptors. 

•  the  characteristics  of  the  1S_A  and  PART_OF  relationships  provide  an  implicit 
mechanism  for  combining  inter-image  motion  descriptions  into  sequence  spanning 
descriptions. 
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A  knowledge  base  of  general  motion  concepts  was  presented  using  this  for¬ 
malism.  Each  concept  was  represented  using  a  frame  whose  slots  reflected  the  seman¬ 
tic  components  necessary  for  adequate  description  of  the  motion.  In  addition,  using 
the  general  motion  terms,  a  problem  specific  knowledge  base  was  shown  for  human  left 
ventricular  segmental  wall  motion  -  both  normal  and  abnormal. 

This  overall  knowledge  base  is  used  for  recognition  of  the  concepts  in  it.  A 
control  structure  was  presented  that  is  based  on  an  integration  of  the  paradigms  of 
hypothesize-and-test,  competition  and  co-operation  among  local  hypotheses  and  feed¬ 
back  systems.  Components  of  the  structure  were  discussed  and  examples  of  both  ac¬ 
tual  and  hypothesized  function  presented.  The  control  structure  possesses  several 
unique  characteristics: 

•  The  set  of  active  hypotheses  is  organized  using  their  conceptual  adjacencies. 

That  is,  hypotheses  co-operate  with  one  another  between  levels  of  abstraction  (IS _ A 

and  PART_OF  relationships)  and  compete  against  one  another  if  they  provide 
conflicting  descriptions  of  motions  for  the  same  time  interval  (similarity  and  "next”  re¬ 
lations). 

•  Hypotheses  are  rated  using  certainty  factors.  The  certainty  factor  is  updated 
after  each  image  using  the  conceptual  adjacency  and  matching  progress  of  it  for  that 
image.  The  updating  rule  is  based  on  relaxation  labelling  methods.  Propagation  of 
changes  in  certainty  factors  is  handled  by  the  relaxation  process  itself. 

•  Temporal  segmentation  is  required  in  order  to  determine  when  a  motion  ends 
and  when  the  next  one  begins.  The  time  boundary  between  two  motions  is  determined 
by  the  maximal  certainty  difference  between  hypotheses  in  their  interval  of  competi¬ 
tion. 

•  Exception  generation  and  analysis  of  the  similarity  links  in  the  parent  hy¬ 
pothesis  form  the  basis  of  the  change  of  attention  mechanism  for  adding  new  hy¬ 
potheses  into  consideration. 

•  A  prediction  guided  edge  finder  using  relaxation  labelling  modified  to  account 
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for  expected  search  window  and  edge  orientation  was  presented. 

•  Inter-image  motions  are  coded  into  the  essential  kineses  of  the  object  using  a 
small  set  of  primitive  motion  types. 

•  Hypothesis  inertia  is  the  guiding  principle  in  the  focus  of  attention  mechanism. 
Using  this  principle,  the  problems  of  erratically  shifting  foci  and  false  peaks  are  greatly 
reduced. 

All  of  the  above  were  integrated  into  a  feedback  loop  (based  on  similar  con¬ 
cepts  in  control  systems)  and  feedback  occurs  at  both  the  image  and  conceptual  lev¬ 
els. 

In  addition  to  the  above,  we  have  shown  that  for  a  scheme  such  as  the  one 
presented,  a  time  constant  can  be  defined  that  provides  a  lower  bound  on  the  number 
of  images  in  which  a  motion  must  be  present  in  the  image  sequence  if  it  is  to  be  recog¬ 
nized.  The  absolute  lower  limit  was  shown  to  be  3  images  in  noisefree  conditions.  We 
also  demonstrated  that  an  independent  temporal  segmentation  process  must  be  used 
if  the  response  time  of  the  "next"  constraint  presents  a  significant  error  to  the  dura¬ 
tion  calculations.  The  lower  bound  on  this  response  time  is  6  images  in  noisefree  con¬ 
ditions.  Finally,  an  empirical  relationship  was  presented  that  relates  hypothesis  ex¬ 
pected  duration,  minimum  number  of  images  required  for  hypothesis  discrimination, 
number  of  simultaneously  competing  hypotheses,  image  noise  characteristics,  and 
scene  sampling  rate.  This  relationship  can  be  used  to  determine  appropriate  sampling 
rates  for  domains  whose  motions  are  represented  in  a  KB,  or  to  suggest  changes  in  KB 
structure  in  order  to  accomodate  the  available  sampling  rate. 

There  are  several  directions  for  future  research: 

•  A  current  limitation  is  that  the  description  level  cannot  be  more  specific  than 

the  user  provides  in  his  initial  guess  to  the  system.  Indeed,  if  he  guessed  wrong,  the 
level  may  be  far  less  specific  than  is  useful  to  him.  Solution  of  this  problem  would  in¬ 
volve  use  of  the  IS_A  hierarchy  after  alternates  are  activated  using  the  similarity  links. 
Hypothesis  activation  would  be  propagated  downwrard  along  IS _ A  links  to  appropriate 
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more  specific  concepts. 

•  Shape  matching  should  be  more  formal  in  nature,  perhaps  using  a  scheme  such 
as  in  [Davis77]  modified  to  account  for  predictions  of  shape  changes. 

•  Shape  change  needs  more  consideration.  However,  this  is  a  very  difficult  con¬ 
cept  to  deal  with  and  the  only  useful  solutions  may  be  very  domain  dependent. 

•  The  representation  formalism,  although  capable  of  representing  any  concept 
tried  thus  far,  is  quite  tedious  to  work  with.  For  example,  in  describing  the  ischemic 
left  ventricle,  we  can  use  the  normal  case  and  provide  a  description  in  terms  of  the 
normal  case.  The  representation  should  be  extended  to  allow  for  perturbations  of  one 
frame  in  order  to  create  new  ones  perhaps  by  the  creation  of  a  new  relation  such  as  a 
MODIFY  primitive.  This  would  also  involve  propagation  of  such  perturbations  along  the 
PART_OF  axis. 

•  Of  course,  a  complete  prototype  implementation  is  essential.  We  currently 
faced  severe  restrictions  on  the  PDP  11/45  we  were  working  on.  Transferring  to  a 
different  machine  is  necessary  at  least  for  the  high  level  functions. 

•  A  formal  mathematical  study  of  the  relationships  between  the  varying  values  of 
compatibility  factors  and  system  response  time,  inter-image  minimum  time  intervals, 
noise  sensitivity  ,  etc.,  should  be  undertaken. 

This,  as  all  other  similar  research  projects,  has  been  an  exercise  in  experi¬ 
mental  science.  The  experimentation  has  been  along  two  separate  lines:  knowledge 
based  vision  systems  and  medical  science.  Our  progress  and  results  have  strongly  sup¬ 
ported  our  methodology.  At  this  point,  we  firmly  believe  that  this  framework  is  indeed 
appropriate  as  a  solution  to  the  motion  understanding  problem  in  specific  types  of 
domains. 
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APPENDIX  A 


MOTION  FRAME  SYNTAX 

The  following  is  the  syntax  used  for  the  frame  definitions  presented  in  this 
thesis.  Upper  case  denotes  non-terminal  symbols  while  lower  case  bold  italics  are 
language  keywords.  The  symbols  "[  ]  ;  ,  <-  are  terminal  symbols.  Additional  terminal 
symbols,  that  do  not  appear  below',  are  introduced  during  discussion  of  the  rules. 
Brace  brackets  "j$"  surround  optional  constructs  while  the  symbol  ”|"  separates  alter¬ 
nates  in  the  production  rule.  Comments  regarding  the  semantics  of  the  production 
rules  are  provided  as  needed. 

FRAME  <=  frame  FRAME_ID  |ISA_EXPR)  with  [PREREQS)  [DEPENDS)  [SIMLINKS)  end 
The  rule  above  permits  a  frame  to  be  empty.  Empty  frames  are  useful  for  or¬ 
ganizing  an  IS_A  hierarchy  and  are  used  for  this  purpose  in  Chapter  4.  FRAME_ID  is  a 
simple  identifier  that  is  unique  within  the  knowledge  base.  Simple  identifiers  are 
strings  of  alphabetic  and  numeric  characters,  along  with  the  underscore  and  be¬ 

ginning  with  an  alphabetic  character. 

PREREQS  <=  prerequisites  PSLOT_LIST 

DEPENDS  <=  dependents  DSL0T_L1ST 

SIMLINKS  <=  similarity  links  SIM_L1ST 

1SA_EXPR  <=  is_a  FRAME_ID_LIST 

A  frame  may  have  several  IS _ A  parents.  In  such  situations  it  is  important  to 

distinguish  between  slots  with  the  same  name  that  are  inherited  from  IS _ A  parent 
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frames. 


SIMILARITY  <=  SIMSEAD  [ SIMILARITY_EXPR}  D1FFERENCE_EXPR  [BINDINGSCPR} 
SIM_HEAD  <=  LINK_ID  :  FRAME_ID 

The  LINK__ID  is  a  simple  identifier  and  must  be  unique  within  the  frame. 

SIMI LARITY_EXP R  <=  •with  similarities  :  SIMIL_LIST  ; 

DIFFERENCE_EXPR  <=  for  differences  :  D1FF_LIST  ; 

BINDING_EXPR  <=  bound  by  :  BIND_L1ST  ; 

PSLOT  <=  SLOT_ID  :  [FRAMES}  [  such  that  [  CONSTRAINTS  ST  ]  } 

A  SLOT_ID  is  a  simple  identifier  and  must  be  unique  within  the  frame.  Note 

that  if  it  is  the  same  as  a  SL0T_ID  in  an  IS _ A  parent  frame,  the  FRAME_ID,  (that  is,  its 

type  )  may  be  omitted  because  it  is  inherited  from  the  parent  frame.  If  it  is  included 
it  must  be  a  sub-class  (with  respect  to  1S__A)  of  the  type  in  the  parent  frame. 

DSLOT  <=  SLOTS  :  [FRAMES}  unth  CALCSST 

SIMIISIST  <=  L0GICAL_EXPR_L1ST 

CALC  <=  SLOT _ ASSIGN  j  such  that  [  CONSTRAINTSST  ]} 

DIFF  <=  DIFFS  :  EXCEPTIONS  uihere  [  LOGICAISXPRSST  ] 

An  EXCEPTION_ID  is  a  simple  identifier  and  corresponds  to  the  exception  type 
as  discussed  in  Chapter  4.  The  L0G1CAL_EXPR_LIST  specifies  constraints  so  that  an  ac¬ 
ceptable  exception  token  may  be  identified. 

BINDSST  <=  SLOT_ASSIGN  j  BINDSST  ,  SL0T_ASS1GN 

SL0T_ASS1GN  <=  SLOTS  «-  ASSIGNMENT_EXPR 

An  ASS1GNMENT_EXPR  is  a  single-valued  expression.  Its  value  may  be  a  frame 


-  144  - 


token,  or  an  arithmetic  quantity.  Arbitrary  arithmetic  expressions  using  the  standard 
operators  for  addition,  subtraction,  multiplication,  division  and  exponentiation,  as  well 
as  trigonometric  functions  may  be  included.  Specific  values  of  slots  are  referred  to  us¬ 
ing  a  complex  identifier.  Complex  identifiers  are  groups  of  simple  identifiers  delimited 
by  periods  In  this  way,  a  path  is  specified  through  the  PART— .OF  hierarchy  so  that 

the  desired  value  may  be  obtained.  The  simple  identifiers  refer  to  slot  names  within 
frames  and  are  specified  placing  the  most  distantly  removed  slot  first.  Note  that 
operations  on  frame  tokens  are  not  defined.  A  special  type  of  assignment  is  the  "de¬ 
fault"  value.  Such  a  value  is  preceeded  by  the  keyword  default  which  follows  the  slot 
name. 

EXCEPTION  <=  exception  [  EXCEPT10N_ID  -with  SL0T_ASS1GN_L1ST  ] 

CONSTRAINT  <=  L0G1CAL-EXPR  [EXCEPTION! 

|  for  SLOT_ID  :  FRAME_ID  where  [  CONSTRAINT_LIST  ] 

The  for  construct  is  used  in  order  to  find  specific  frame  tokens  to  which  the 
current  slot  is  to  be  related.  The  CONSTRAINT_LIST  in  this  construct  specifies  both 
specific  values  that  this  token  must  have  in  order  for  it  to  be  acceptable  as  well  as  rela¬ 
tionships  of  the  token’s  slots  to  the  current  slot. 

LOGICAL—EXPR  <=  BOOLEAN-FUNCTION  [  at  ASS1GNMENT_EXPR} 

BOOLEAN_FUNCTION  is  any  expression  that  may  be  formed  using  the  and  ,  or 
and  ~  (negation)  operators  as  well  as  parentheses  along  with  complex  identifiers 
and  numerical  comparison  operators  "<,>,  =  ,~=,^,^".  Single-valued  arithmetic  expres¬ 
sions  may  also  be  included.  The  at  construct  is  used  to  specify  the  time  instant  at  (or 
interval  during)  which  the  boolean  expression  must  be  true.  Therefore,  the 
ASSIGNMENT_EXPR  must  evaluate  to  a  single  value  of  time  dimension,  or  to  a  token  of 
type  TIME-INTERVAL.  Operands  are  either  identifiers  or  function  calls  that  return  sin¬ 
gle  values  or  binary  relationships  such  as  "instance_of"  or  "part_of".  The  negation 
operator  may  preceed  any  of  these  in  order  to  test  for  the  negation  of  the  relationship. 
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Note  that  it  is  useful  to  also  allow  the  at  construct  mentioned  above,  to  be  attached  to 
any  identifier  or  function  call.  This  permits  comparison  of  arbitrary  items  through 
time. 

SL0T__ASS1GN_LIST  <=  SLOT_ASSIGN  |  SL0T_ASSIGN__L1ST  ,  SLOT__ASSIGN 
L0G1CAL_EXPR_LIST  <=  LOGICAL_EXPR  |  L0G1CA1^EXPR_LIST  ,  LOGICAL_EXPR_LIST 
FRAM E_ID_L1  ST  <=  FRAME_ID  j  FRAME_ID_LIST  ,  FRAME_ID 
PSL0T_L1ST  <=  PSLOT  ;  |  PSL0T_L1ST  PSLOT  ; 

DSL0T_L1ST  <=  DSLOT  ;  |  DSL0T_L1ST  DSLOT  ; 

SIM_L1ST  <=  SIMILARITY  ;  SIM_L1ST  SIMILARITY  ; 

CALC_LIST  <=  CALC  j  CALC_LIST  ,  CALC 
DIFF_L1ST  <=  DIFF  j  DIFF_L1ST  ,  DIFF 

C0NSTRAINT_L1ST  <=  CONSTRAINT  I  CONS TRAI NT_LI ST  ,  CONSTRAINT 
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APPENDIX  B 


THE  FRAMES  FOR  THE 
CENTRAL  APICAL  ISCHEMIC  LV  CYCLE 


frame  1SCHEM1C_CAP_LV_CYCLE  is_a  1SCHEMIC_LV_ CYCLE  with 
prerequisit  es 

subj  :  ISCK_CAP_LV  ; 
systole  :  1SCH_CAP_SYST0LE  such  that 
[  systole_time_int.end_time  = 

diastole. time_int.start_time  , 
systole. subj  =  self.subj  ]; 
diastole  :  1SCH__CAP_D1AST0LE  such  that 
[  diastole. subj  =  self.subj  ]; 
dependents 

time_int  :  with 

start_time  <-  systole. time_int.start_time  , 
end_time  <-  diastole. time_int. end _ time  ; 

similarity  links 

sim_link  1  :  N_LV_ CYCLE 
for  differences  : 

dl  :  NO_ISCH_CAP_  SYSTOLE  where 

[  prereq_part  instance_of  N_SYSTOLE  ], 
d2  :  N 0_ISCH_CAP_D1  ASTOLE  where 

[  prereq_part  instance_of  N__DIASTOLE  ]; 

bound  by  : 

subj  -*  subj  ;; 

end 
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frame  ISCH_CAP_SYSTOLE  xs_a  ABN_SYSTOLE  with 
prerequisit  es 

subj  :  ISCH_CAP_LV  ; 

isovoL_contract  :  N_1S0V0LUMIC_C0NTRACTI0N  such  that 

[  isovo] _ contract. subj  =  self.subj  , 

isovoLcontract.time_int.end_.time  = 

max_ejection.time_int.start_time  ]; 

max_ejection  :  1SCH_CAP_MAXIMUM _ EJECTION  such  that 

[  max_ejection.subj  =  self.subj  , 

max_ejection.time_int.  end _ time 

=  red _ ejection. time_int.start_time  ]; 

red _ ejection  :  ISCK_CAP_REDUCED_EJECTION  such  that 

[  red _ ejection. subj  =  self.subj  ]; 

dependents 

time_int  :  with 

start_time  «-  isovol_contract.time_int.start_time  , 

end _ time  *-  red _ ejection. time_int.end _ time  ; 

similarity  links 

sim _ linkl  :  N_SYST0LE 

for  differences  : 

dl  :  T00_FAST  where 

[  subj  instance_of  ISCK_CAP_SEG  , 
time_int  =  max_ejection.time_int  ], 
d2  :  T00_FAST  where 

[  subj  instance_of  ISCH_CAP_SEG  , 
time_int  =  red_ejection.time_int  ]; 
bound  by  : 

subj  ->  subj  ;; 

end 
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frame  ISCH_CAP_D1AST0LE  ts_ a  ABN_D1AST0LE  with 
prerequisit  es 

subj  :  1SCH_CAP_LV  ; 

isovoL_relax  :  ISCH_CAP_ISOVOLUMIC_RELAXATION  such  that 

[  isovol _ relax.time_int.end _ time  = 

rapid_fill.time_int.start_time  , 
isovol_relax.subj  =  self.subj  ]; 
rapid_fill  :  1SCH_CAP_RAPID_FILL  such  that 

[  rapid _ fill. subj  =  self.subj  , 

rapid _ fill.time_int.end_time 

=  diastasis. time_int.start_time  ]; 
diastasis  :  ISCH_CAP_D1ASTAS1S  such  that 

[  diastasis. time_int. end _ time 

=  filL_atrial_cont.time_int.start_time  , 
diastasis. subj  =  self.subj  ]; 

fill _ atrial _ cont  :  ISCH_CAP_FILL_ATR1AL_C0NT  such  that 

[  filL_atrial_cont.subj  =  self.subj  ]; 
dependents 

time_int  :  with 

start_time  *-  isovol _ relax. time_int.start_time  , 

end_time  *-  fill _ atrial _ cont. time_int. end _ time  ; 

similarity  links 

sim _ linkl  :  N_DIASTOLE 

for  differences  : 

dl  :  N0_1SCH_CAP_IS0V0LUMIC_RELAXAT10N  where 

[  prereq_part  instance_of  N_1S0V0LUM1C_RELAXAT10N  ], 
d2  :  TOO_FAST  where 

[  subj  instance_of  ISCH_CAP_SEG  , 

time_int  =  rapid _ fill.time_int  ], 

d3  :  TOO_FAST  where 

[  subj  instance_of  1SCH_CAP_SEG  , 
time_int  =  diastasis. time_int  ], 
d4  :  TOO_FAST  where 

[  subj  instance_of  ISCH_CAP_SEG  , 

time_int  =  fill _ atrial _ cont.time_int  ]; 

hound  by  : 

subj  ->  subj  ;; 

end 
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frame  ISCH_CAP_ISOVOLUMIC_RELAXATION  is_a  ABN_1S0V0LUMIC_RELAXATI0N 

urith 

prere  quisites 

subj  :  ISCH_CAP_LV  ; 

anterior_mot  :  N_ANT_ISO_RELAX  such  that 
[  anterior_mot.subj  part_of  self.subj  ]; 

apical _ mot  :  ISCH _ CAP_ISO_RELAX  such  that 

[  apical _ mot. subj  part_of  self.subj  ]; 

posterior_mot  :  N_POST_ISO_RELAX  such  that 
[  posterior_mot.subj  part_of  self.subj  ]; 
dependents 

time_int  :  urith 

start_time  «-  anterior_mot.time_int.start_time  such  that 
[  same(anterior_mot.time_int.start_time  , 

apical _ mot.time__int.start_time  , 

posterior_mot.time_int.start_time  )  ], 

end _ time  <-  anterior_mot.time_J.nt.end_time  such  that 

[  same(anterior_mot.time_int.end _ time  , 

apical _ mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ]; 

similarity  links 

sim _ linkl  :  N_1S0V0LUMIC_RELAXATI0N 

for  differences  : 

dl  :  TOO_LITTLE_M 0T10N  where 

[  subj  instance_of  1SCH_CAP_SEG  ]; 

bound  by  : 

anterior_mot  -*  anterior_mot  , 
posterior_mot  -»  posterior_mot  ;; 

end 
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frame  ISCH__CAP_1SQ_RELAX  is__ a  ABN_ISO_RELAX  -with 

prerequisites 

subj  :  1SCH_CAP_SEG  ; 

translate  :  TRANSLATE  such  that 
[  translate. subj  =  self.subj  , 

((  for  dirl  :  TOWARDS  where 
[  dirl. subj  =  self.subj  , 
dirl. ref  =  super__part(self).  centroid  , 
translate. speed  >  2  /  dir l.time_int. duration  ])  or 
(  for  dir2  :  AWAY_FROM  where 
[  dir2.subj  =  self.subj  , 
dir2.ref  =  super_part(se If). centroid  , 
translate. speed  >  6  /  dir2.time_int. duration  ]  ) 
exception  T00_L1TTLE_M0T10N  with 
[  subj  <-  self.subj  ]  ]  ; 

dependents 

time__int  :  with 

start_time  <-  translate. time_int.start_time  , 
end _ time  <-  translate. ti_me_int. end _ time  ; 

end 
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frame  ISCH_CAP_MAXIMUM_EJECTION  is_a  ABN  MAXIMUM  EJECTION  with 
prerequisites 

subj  :  1SCH_CAP_LV  ; 

anterior_mot  :  N_ANT_MAX_EJ  such  that 
[  anterior_mot.subj  part_of  self. subj  , 
for  dirl  :  TOWARDS  inhere 

[  dirl. subj  =  anterior _ mot. subj  , 

dirl. ref  =  self.subj. centroid  ]]; 
apical_mot  :  1SCH_CAP_MAX_EJ  such  that 

[  apical _ mot. subj  part__of  self.subj  , 

for  dir 2  :  TOWTARDS  -where 

[  dir2.subj  =  apical _ mot. subj  , 

dir2.ref  =  self.subj. centroid  ]. 

(apical _ mot. translate. speed  <  anterior_mot. translate. speed  and 

apical mot. translate. speed  <  posterior__mot. translate. speed  )  and 

posterior_mot.  translate,  speed  =  anterior_mot.  translate,  speed 
exception  TOO_FAST  with 

[  subj  <-  1SCH_CAP_SEG  , 
time_int  <-  self.time_int  ]]; 
posterior_mot  :  N_POST_MAX_EJ  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir 3  :  TOWARDS  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self.subj. centroid  ]]; 


dependents 

time_int  :  with 

start_time  «-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 

apical _ mot.time_int.start_time  , 

posterior_mot.time_int.start_.time  )  ], 

end _ time  *-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical_mot.time_int.end_time  , 
posterior_mot.time_int.encL_time  )  ]; 

end 
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frame  ISCH_CAP_REDUCED_EJECTION  is_a  1SCH_REDUCED_EJECT10N  with 
prerequisxt  es 

subj  :  ISCH_CAP_LV  ; 

anterior_mot  :  N_ANT_RED_EJ  such  that 
[  anterior_mot.subj  part_of  self.subj  , 
for  dirl  :  TOWARDS  where 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self.subj. centroid  ]]; 

apical _ mot  :  1SCH_CAP_RED_EJ  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

for  dir 2  :  TOWARDS  where 

[  dir2.subj  =  apical_mot.subj  , 
dir2.ref  =  self. subj. centroid  ], 

(apical _ mot. translate. speed  <  anterior_mot.translate. speed  and 

apical_mot. translate. speed  <  posterior_mot. translate. speed  )  and 
posterior_mot. translate. speed  =  anterior_mot. translate,  speed 
exception  TOO_FAST  with 

[  subj  -  ISCH_CAP_SEG  .  _ 
time_int  <-  self.time_int  ]]; 
posterior_mot  :  N_POST_RED_EJ  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir3  :  TOWARDS  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self.subj. centroid  ]]; 


dependents 

time_int  :  with 

start_time  <-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start__time  , 
apical_mot.time_int.start_time  , 
posterior_mot.time_int.start_time  )  ], 

end _ time  «-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ]; 

end 
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frame  ISCH_CAP_RAPID_F1LL  is_ a  ABN_RAPID_FILL  with 
prerequisites 

subj  :  ISCH_CAP_LV  ; 

anterior_mot  :  N_ANT_RAP_FILL  such  that 
[  anterior_mot.subj  part_of  self.subj  , 
for  dirl  :  AWAY_FROM  where 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self.subj. centroid  ]]; 
apical_mot  :  ISCH_CAP_RAP_FILL  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

for  dir2  :  AWAY_FROM  where 

[  dir2.subj  =  apical_mot.subj  . 
dir2.ref  =  self.subj. centroid  ], 

(apical mot. translate. speed  <  anterior_mot. translate. speed  and 

apical _ mot. translate. speed  <  posterior_mot. translate. speed  )  and 

posterior_mot.  translate,  speed  =  anterior_mot.  translate,  speed 
exception  TOO_FAST  with 

[  subj  «-  1SCH_CAP_SEG  , 
time_int  «-  self.time_int  ]]; 
posterior_mot  :  N_POST_RAP_FILL  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir 3  :  AWAY_FROM  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self. subj. centroid  ]]; 


dependents 

time_int  :  with 

start_time  «-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 
apical_mot.time_int.start_time  , 
posterior_mot.time_int.start_time  )  ]  , 

end _ time  <-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ]  ; 

end 
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frame  1SCH_CAP_D1ASTAS1S  is__a  ABN_D1ASTASIS  with 


prerequisites 

subj  :  1SCH_CAP_LV  ; 

anterior_mot  :  N__ANT_DIAS  such  that 

[  anterior_mot.subj  part_of  self. subj  , 
for  dirl  :  AWAY_FROM  -where 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self.subj. centroid  ]]; 


apical_mot  :  ISCH_CAP_D1AS  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

for  dir2  :  AWAY_FROM  where 

[  dir2.subj  =  apical_mot.subj  , 
dir2.ref  =  self.subj. centroid  ], 

(apical _ mot. translate,  speed  <  anterior_mot. translate. speed  arid. 

apical_mot. translate. speed  <  posterior_mot. translate. speed  )  and 
posterior_mot. translate. speed  =  anterior_mot.  translate,  speed 
exception  TOCL.FAST  with 

[  subj  <-  1SCH_CAP_SEG  , 
time_int  <-  self.time_int  ]]; 
posterior_mot  :  N_POST_DIAS  such  that 

[  posterior _ mot. subj  part_of  self.subj  , 

for  dir 3  :  AWAY_FROM  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self.subj. centroid  ]]; 


dependents 

time_int  :  with 

start_time  <-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 
apical_mot.time_int.start_time  , 
posterior_mot.time_int.start_time  )  ], 

end _ time  <-  anterior_mot.time_int.end _ time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical_mot.time_int.end _ time  , 

posterior_mot.time_int.end_time  )  ]; 

end 
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frame  1SCH_CAP_FILL_ATRIAL_C0NT  is^a  ABN_FILLt_ATRIAIl_CONT  with 
prerequisites 

subj  :  ISCH_CAP_LV  ; 

anterior_mot  :  N_ANT_FILL_ATR_CONT  such  that 
[  anterior_mot.subj  part_of  self.subj  , 
for  dirl  :  AWAY_FROM  inhere 

[  dirl. subj  =  anterior_mot.subj  , 
dirl. ref  =  self.subj. centroid  ]]; 

apical _ mot  :  1SCH _ CAP _ FILL_ATR_CONT  such  that 

[  apical _ mot. subj  part_of  self.subj  , 

for  dir2  :  AWAY_FROM  inhere 

[  dir2.subj  =  apical _ mot. subj  , 

dir2.ref  =  self.subj. centroid  ], 

(apical _ mot. translate. speed  <  anterior_mot. translate. speed  and 

apical _ mot. translate. speed  <  posterior_mot. translate. speed  )  and 

posterior_mot. translate. speed  =  anterior_mot.  translate,  speed 
exception  TOO_FAST  with 

[  subj  <-  1  SCK_CAP_SEG  , 
time_int  *-  self.time_int  ]]; 
posterior_mot  :  N_POST_FILL_ATR_CONT  such  that 
[  posterior_mot.subj  part_of  self.subj  , 
for  dir 3  :  AWAY_FROM  where 

[  dir3.subj  =  posterior_mot.subj  , 
dir3.ref  =  self.subj. centroid  ]]; 

dependents 

time_int  :  with 

start_time  «-  anterior_mot.time_int.start_time 

such  that 

[  same(anterior_mot.time_int.start_time  , 
apical_mot.time_int.start_time  , 
posterior_mot.time_int.start_time  )  ], 

end _ time  <-  anterior_mot.time_int.end_time  such  that 

[  same(  anterior_mot.time_int.end _ time  , 

apical _ mot.time_int.end _ time  , 

posterior_mot.time_int.end _ time  )  ]; 

end 
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Data  Base  Conference,  1975] 

CSRG-57  MERLIN:  TOWARDS  AN  IDEAL  PROGRAMMING  LANGUAGE 
Eric  C.R.  Hehner,  July  1975 

see  Acta  Informatica  Col.  10,  No. 3,  pp. 229-243,  1978 

CSRG-58  ON  THE  SEMANTICS  OF  THE  RELATIONAL  DATA  MODEL 
Hans  Albrecht  Schmid  and  J.  Richard  Swenson, 

July  1975  [Proceedings  of  the  ACM  S1GM0D  Conference,  1975] 

CSRG-59  THE  SPECIFICATION  AND  APPLICATION  TO  PROGRAMMING 
OF  ABSTRACT  DATA  TYPES 
John  V.  Guttag,  September  1975 
[Ph.D.  Thesis,  DCS,  1975] 

CSRG-60  NORMALIZATION  AND  FUNCTIONAL  DEPENDENCIES  IN  THE 
RELATIONAL  DATA  BASE  MODEL 
Phillip  Alan  Bernstein,  October  1975 
[Ph.D.  Thesis,  DCS,  1975] 

CSRG-61  LSL:  A  LINK  AND  SELECTION  LANGUAGE 

r 

D.  Tsichritzis,  November  1975  [Proceedings  ACM 
S1GM0D  Conference,  1976] 


*  CSRG-62  COMPLEMENTARY  DEFINITIONS  OF  PROGRAMMING  LANGUAGE 

SEMANTICS 

James  E.  Donahue,  November  197b 
[Ph.D.  Thesis,  DCS,  1975] 

CSRG-63  AN  EXPERIMENTAL  EVALUATION  OF  CHESS  PLAYING  HEURISTICS 
Lazio  Sugar,  December  1975 
[M.Sc.  Thesis,  DCS,  1975] 

CSRG-64  A  VIRTUAL  MEMORY  SYSTEM  FOR  A  RELATIONAL  ASSOCIATIVE 
PROCESSOR 

S.A.  Schuster,  E.A.  OzKarahan,  and  K.C.  Smith, 

February  1976  [Proceedings  National  Computer 
Conference  1976,  v.45,  pp. 855-362] 

CSRG-65  PERFORMANCE  EVALUATION  OF  A  RELATIONAL  ASSOCIATIVE 
PROCESSOR 

E.A.  Ozkarahan,  S.A.  Schuster,  and  K.C.  Sevcik, 

February  1976  [ACM  Transactions  on  Database 
Systems,  v.  1,  n:4,  December  1976] 

CSRG-66  EDITING  COMPUTER  ANIMATED  FILM 
Michael  D.  Tilson,  February  1976 
[M.Sc.  Thesis,  DCS,  1975] 

CSRG-67  A  DIAGRAMMATIC  APPROACH  TO  PROGRAMMING  LANGUAGE 
SEMANTICS 

James  R.  Cordy,  March  1976 
[M.Sc.  Thesis,  DCS,  1976] 

*  CSRG-68  A  SYNTHETIC  ENGLISH  QUERY  LANGUAGE  FOR  A  RELATIONAL 

ASSOCIATIVE  PROCESSOR 

L.  Kerschberg,  E.A.  Ozkarahan,  and  J.E.S.  Pacheco, 

April  1976 

CSRG-69  AN  ANNOTATED  BIBLIOGRAPHY  ON  COMPUTER  PROGRAM 
ENGINEERING 

D.  Barnard  and  D.  Thompson  (eds.),  Fourth  Edition, 

May  1976 

*  CSRG-70  A  TAXONOMY  OF  DATA  MODELS 

L.  Kerschberg,  A.  Klug,  and  D.Tslchrit.zis,  May  1976 
[Proceedings  Very  Large  Data  Base  Conference,  1976] 

*  CSRG-71  OPTIMIZATION  FEATURES  FOR  THE  ARCHITECTURE  OF  A 

DATA  BASE  MACHINE 

E. A.  Ozkarahan  and  K.C.  Sevcik,  May  1976 

[ACM  Transactions  of  Database  Systems,  v.2,  n.4,  December  1977] 

*  CSRG-72  THE  RELATIONAL  DATA  BASE  SYSTEM  OMEGA  -  PROGRESS  REPORT 

H.A.  Schmid  (ed.),  P.A.  Bernstein  (ed.),  B.  Arlow, 

R.  Baker  and  S.  Pozgaj,  July  1976 
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CSRG-73  AN  ALGORITHMIC  APPROACH  TO  NORMALIZATION  OF 
RELATIONAL  DATA  BASE  SCHEMAS 
P.A.  Bernstein  and  C.  Beeri,  September  1976 

*  CSRG-74  A  HIGH-LEVEL  MACHINE-ORIENTED  ASSEMBLER  LANGUAGE 
FOR  A  DATA  BASE  MACHINE 

E.A.  Ozkarahan  and  S.A.  Schuster,  October  1976 

CSRG-75  DO  CONSIDERED  OD:  A  CONTRIBUTION  TO  THE  PROGRAMMING 
CALCULUS 

Eric  C.R.  Kehner,  November  1976 
Acta  Informatica  to  appear  1979 

CSRG-76  SOFTWARE  HUT:  A  COMPUTER  PROGRAM  ENGINEERING 
PROJECT  IN  THE  FORM  OF  A  GAME 
J.J.  Horning  and  D.B.  Wortman,  November  1976 

[IEEE  Transactions  on  Software  Engineering,  v.SE-3,  n.4,  July  1977] 

CSRG-77  A  SHORT  STUDY  OF  PROGRAM  AND  MEMORY  POLICY  BEHAVIOUR 
G.  Scott  Graham,  January  1977 

CSRG-78  A  PANACHE  OF  DBMS  IDEAS 

D.  Tsichritzis  (ed.),  February  1977 

CSRG-79  THE  DESIGN  AND  IMPLEMENTATION  OF  AN  ADVANCED  LALR 
PARSE  TABLE  CONSTRUCTOR 
David  K.  Thompson,  April  1977 
[M.Sc.  Thesis,  DCS,  1976] 

CSRG-80  AN  ANNOTATED  BIBLIOGRAPHY  ON  COMPUTER  PROGRAM 
ENGINEERING 

D.  Barnard  (ed.),  Fifth  Edition,  May  1977 

CSRG-81  PROGRAMMING  METHODOLOGY:  AN  ANNOTATED  BIBLIOGRAPHY 
FOR  IFIP  WORKING  GROUP  2.3 

Sol  J.  Greenspan  and  J.J.  Horning  (eds.).  First  Edition,  May  1977 
CSRG-82  NOTES  ON  EUCLID 

edited  by  W.  David  Elliot  and  David  T.  Barnard,  August  1977 

CSRG-B3  TOPICS  IN  QUEUEING  NETWORK  MODELING 
edited  by  G.  Scott  Graham,  July  1977 

CSRG-S4  TOWARD  PROGRAM  ILLUSTRATION 

Edward  Yarvvood,  September  1977 
[M.Sc.  Thesis,  DCS,  1974] 

CSRG-85  CHARACTERIZING  SERVICE  TIME  AND  RESPONSE  TIME 

DISTRIBUTIONS  IN  QUEUEING  NETWORK  MODELS  OF  COMPUTER 
SYSTEMS 

Edward  D.  Lazowska,  September  1977 
[Ph.D.  Thesis,  DCS,  1977] 
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CSRG-8S  MEASUREMENTS  OF  COMPUTER  SYSTEMS  FOR  QUEUEING 
NETWORK  MODELS 
Martin  G.  Kienzie,  October  1977 

[M.Sc.  Thesis,  DCS,  1977;  Proc.  Int.  Symp.  on  Modelling  and  Performance 
Evaluation  of  Computer  Systems,  Vienna,  1979] 

CSRG-87  ’OLGA’  LANGUAGE  REFERENCE  MANUAL 

B.  Abourbih,  H.  Trickey,  D.M.  Lewis,  E.S.  Lee, 

P.l.P.  Boulton,  November  1977 

CSRG-88  USING  A  GRAMMATICAL  FORMALISM  AS  A  PROGRAMMING  LANGUAGE 
Brad  A.  Silverberg,  January  1978 
[M.Sc.  Thesis,  DCS,  1978] 

CSRG-89  ON  THE  IMPLEMENTATION  OF  RELATIONS:  A  KEY  TO  EFFICIENCY 
Joachim  W.  Schmidt,  January  1978 

CSRG-90  DATA  BASE  MANAGEMENT  SYSTEM  USER  PERFORMANCE 
Frederick  H.  Lochovsky,  April  1978 
[Ph.D.  Thesis,  DCS,  1978] 

CSRG-91  SPECIFICATION  AND  VERIFICATION  OF  DATA  BASE 
SEMANTIC  INTEGRITY 
Michael  Lawrence  Brodie,  April  1978 
[Ph.D.  Thesis,  DCS,  1978] 

CSRG-92  STRUCTURED  SOUND  SYNTHESIS  PROJECT  (SSSP): 

AN  INTRODUCTION 

by  William  Buxton,  Guy  Fedorkow,  with  Ronald  Baecker, 

Gustav  Ciamaga,  Leslie  Mezei  and  K.C.  Smith,  June  1978 

CSRG-93  A  DEVICE-INDEPENDENT, GENERAL-PURPOSE  GRAPHICS  SYSTEM 
IN  A  MINICOMPUTER  TIME-SHARING  ENVIRONMENT 
William  T.  Reeves,  August  1978 
[M.Sc.  Thesis,  DCS,  1976] 

CSRG-94  ON  THE  AXIOMATIC  VERIFICATION  OF 
CONCURRENT  ALGORITHMS 
Christian  Lengauer,  August  1978 
[M.Sc.  Thesis,  DCS,  1978] 

CSRG-95  PISA:  A  PROGRAMMING  SYSTEM  FOR  INTERACTIVE 
PRODUCTION  OF  APPLICATION  SOFTWARE 
Rudolf  Marty,  August  1978 

CSRG-96  ADAPTIVE  MICROPROGRAMMING  AND  PROCESSOR  MODELING 
Walter  G.  Rosocha 
[Ph.D.  Thesis,  EE,  August  1978] 

CSRG-97  DESIGN  ISSUES  IN  THE  FOUNDATION  OF  A  COMPUTER-BASED 
TOOL  FOR  MUSIC  COMPOSITION 
William  Buxton 

[M.Sc.  Thesis,  CSRG,  October  1978] 
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CSRG-S8  THEORY  OF  DATABASE  MAPPINGS 
Anthony  C.  Klug 

[Ph.D.  Thesis,  DCS,  December  1978] 

CSRG-99  HIERARCHICAL  COROUTINES:  A  MECHANISM  FOR  IMPROVED 
PROGRAM  STRUCTURE 
Leonard  1.  Vanck,  February  1979 

CSRG-10C  TOPICS  IN  PERFORMANCE  EVALUATION 
G.  Scott  Graham  (ed.),  July  1979 

CSRG-101  A  PANACHE  OF  DBMS  IDEAS  II 

F.H.  Lochovsky  (ed.),  May  1979 

CSRG-102  A  SIMPLE  SET  THEORY  FOR  COMPUTING  SCIENCE 
Eric  C.R.  Hehner,  May  1979 

CSRG-103  THE  CENTRALIZED  ALGORITHM  IN  DISTRIBUTED  SYSTEMS 
Ernest  J.H.  Chang 
[Ph.D.  Thesis,  DCS,  July  1979] 

CSRG-1 04  ELIMINATING  THE  VARIABLE  FROM  DIJKSTRA’S 
MINI-LANGUAGE 
D.  Hugh  Redelmeier,  July  1979 

CSRG-1 05  A  LANGUAGE  FACILITY  FOR  DESIGNING  INTERACTIVE 
DATABASE-INTENSIVE  APPLICATIONS 
John  Mylopoulos,  Philip  A.  Bernstein,  Harry  K.T.  Wong, 
July  1979 

CSRG-108  ON  APPROXIMATE  SOLUTION  TECHNIQUES  FOR 

QUEUEING  NETWORK  MODELS  OF  COMPUTER  SYSTEMS 
Satish  Kumar  Tripathi,  July  1979 

CSRG-107  A  FRAMEWORK  FOR  VISUAL  MOTION  UNDERSTANDING 
John  K.  Tsotsos,  John  Mylopoulos,  H.  Dominic  Cowey 
Steven  W.  Zucker,  DCS,  June  1979 

CSRG-108  DIALOGUE  ORGANIZATION  AND  STRUCTURE  FOR 
INTERACTIVE  INFORMATION  SYSTEMS 
John  Leonard  Barron 
[M.Sc.  Thesis,  DCS,  1980] 

CSRG-1 09  A  UNIFYING  MODEL  OF  PHYSICAL  DATABASES 
D.S.  Batory,  C.C.  Gotlieb,  April  1980 

CSRG-1 10  OPTIMAL  FILE  DESIGNS  AND  REORGANIZATION  POINTS 
D.S.  Batory,  April  1980 

CSRG-1 11  A  PANACHE  OF  DBMS  IDEAS  HI 
D.  Tsichritzis  (ed.),  April  1980 
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CSRG-1 12  TOPICS  IN  PSN  -  II:  EXCEPTIONAL  CONDITION 

HANDLING  IN  PSN;  REPRESENTING  PROGRAMS  IN  PSN; 
CONTENTS  IN  PSN 

Yves  Lesperance,  Byran  M.  Kramer,  Peter  F.  Schneider 
April,  1930 

CSRG-1 13  SYSTEM-ORIENTED  MACRO-SCHEDULING 
C.C.  Gotlieb  and  A.  Schonbach 
May  1980 

CSRG-1 14  A  FRAMEWORK  FOR  VISUAL  MOTION  UNDERSTANDING 
John  Konstantine  Tsot.sos 
[Ph.D.  Thesis,  DCS,  June  1980] 


